Original: Energy-Based Contrastive Learning of Visual Representations
Revised: Energy-Based Contrastive Learning of Visual Representations

Original: Abstract
Revised: Abstract

Original: Contrastive learning is a method of learning visual representations by training
Revised: Contrastive learning is a method of learning visual representations by training

Original: Deep Neural Networks (DNNs) to increase the similarity between representations of positive pairs (transformations of the same image) and reduce the similarity between representations of negative pairs (transformations of different images).
Revised: Deep Neural Networks (DNNs) to increase the similarity between representations of positive pairs (transformations of the same image) and reduce the similarity between representations of negative pairs (transformations of different images).

Original: Here we explore Energy-Based Contrastive Learning (EBCLR) that leverages the power of generative learning by combining contrastive learning with EnergyBased Models (EBMs).
Revised: Here we explore Energy-Based Contrastive Learning (EBCLR) that leverages the power of generative learning by combining contrastive learning with EnergyBased Models (EBMs).

Original: EBCLR can be theoretically interpreted as learning the joint distribution of positive pairs, and it shows promising results on small and medium-scale datasets such as MNIST, Fashion-MNIST, CIFAR-10, and CIFAR100.
Revised: EBCLR can be theoretically interpreted as learning the joint distribution of positive pairs, and it shows promising results on small and medium-scale datasets such as MNIST, Fashion-MNIST, CIFAR-10, and CIFAR100.

Original: Specifically, we find EBCLR demonstrates from × 4 up to × 20 acceleration compared to SimCLR and MoCo v2 in terms of training epochs.
Revised: Specifically, we find EBCLR demonstrates from × 4 up to × 20 acceleration compared to SimCLR and MoCo v2 in terms of training epochs.

Original: Furthermore, in contrast to SimCLR, we observe EBCLR achieves nearly the same performance with 254 negative pairs (batch size 128 ) and 30 negative pairs (batch size 16 ) per positive pair, demonstrating the robustness of EBCLR to small numbers of negative pairs.
Revised: Furthermore, in contrast to SimCLR, we observe EBCLR achieves nearly the same performance with 254 negative pairs (batch size 128 ) and 30 negative pairs (batch size 16 ) per positive pair, demonstrating the robustness of EBCLR to small numbers of negative pairs.

Original: Hence, EBCLR provides a novel avenue for improving contrastive learning methods that usually require large datasets with a significant number of negative pairs per iteration to achieve reasonable performance on downstream tasks.
Revised: Hence, EBCLR provides a novel avenue for improving contrastive learning methods that usually require large datasets with a significant number of negative pairs per iteration to achieve reasonable performance on downstream tasks.

Original: Code: https://github.com/1202kbs/EBCLR
Revised: Code: https://github.com/1202kbs/EBCLR

Original:  Introduction
Revised:  Introduction

Original: In computer vision, supervised learning requires a large-scale human-annotated dataset of images to train accurate deep neural networks (DNNs).
Revised: In computer vision, supervised learning requires a large-scale human-annotated dataset of images to train accurate deep neural networks (DNNs).

Original: However, acquiring labels for millions of images can be difficult or impossible in practice.
Revised: However, acquiring labels for millions of images can be difficult or impossible in practice.

Original: This has led to the rise of self-supervised learning, which learns useful visual representations by forcing DNNs to be invariant or equivariant to image transformations.
Revised: This has led to the rise of self-supervised learning, which learns useful visual representations by forcing DNNs to be invariant or equivariant to image transformations.

Original: Among self-supervised learning algorithms, contrastive methods are rapidly gaining popularity for their superb performance.
Revised: Among self-supervised learning algorithms, contrastive methods are rapidly gaining popularity for their superb performance.

Original: Specifically, contrastive learning methods [1, 2, 3, 4, 5] train DNNs by increasing the similarity between representations of positive pairs (transformations of the same image) and decreasing the similarity between representations of negative pairs (transformations of different images).
Revised: Specifically, contrastive learning methods [1, 2, 3, 4, 5] train DNNs by increasing the similarity between representations of positive pairs (transformations of the same image) and decreasing the similarity between representations of negative pairs (transformations of different images).

Original: The negative pairs prevent DNNs from collapsing to the trivial constant function.
Revised: The negative pairs prevent DNNs from collapsing to the trivial constant function.

Original: There are numerous contrastive learning methods, such as SimCLR [4], Momentum Contrast (MoCo) [5], etc.
Revised: There are numerous contrastive learning methods, such as SimCLR [4], Momentum Contrast (MoCo) [5], etc.

Original: Despite this flurry of research in contrastive learning, contrastive methods require large datasets and a large number of negative pairs per positive pair to achieve reasonable performance on downstream tasks.
Revised: Despite this flurry of research in contrastive learning, contrastive methods require large datasets and a large number of negative pairs per positive pair to achieve reasonable performance on downstream tasks.

Original: Although there are recently proposed non-contrastive methods such as BYOL [6] and SimSiam [7] that do not rely on negative pairs, they require heuristic techniques such as stop-gradient to avert collapsing to the trivial solution.
Revised: Although there are recently proposed non-contrastive methods such as BYOL [6] and SimSiam [7] that do not rely on negative pairs, they require heuristic techniques such as stop-gradient to avert collapsing to the trivial solution.

Original: There has been an effort to explain the dynamics of non-contrastive methods with linear neural networks [3], but it is unclear how the analyses generalize to DNNs.
Revised: There has been an effort to explain the dynamics of non-contrastive methods with linear neural networks [3], but it is unclear how the analyses generalize to DNNs.

Original: 𝑣 + 𝑣 −
Revised: 𝑣 + 𝑣 −

Original: Shared Params 𝑣 ∝1/𝑝(𝑣,𝑣 + ) 𝑓 𝜃 𝑓
Revised: Shared Params 𝑣 ∝1/𝑝(𝑣,𝑣 + ) 𝑓 𝜃 𝑓

Original: 𝜃 𝑓 𝜃 𝑧 𝑧 + 𝑧 −
Revised: 𝜃 𝑓 𝜃 𝑧 𝑧 + 𝑧 −

Original: Projection Space 𝑧
Revised: Projection Space 𝑧

Original: + 𝑧 𝑧 − 𝑝 𝑣,𝑣 + ≫ 𝑝(𝑣,𝑣 − )
Revised: + 𝑧 𝑧 − 𝑝 𝑣,𝑣 + ≫ 𝑝(𝑣,𝑣 − )

Original: Projections ∝1/𝑝(𝑣,𝑣 − )
Revised: Projections ∝1/𝑝(𝑣,𝑣 − )

Original: In this paper we explore a novel avenue in visual representation learning: Energy-Based Contrastive
Revised: In this paper we explore a novel avenue in visual representation learning: Energy-Based Contrastive

Original: Learning (EBCLR) which leverages the power of generative learning [8, 9, 10] by combining contrastive learning with energy-based models (EBMs).
Revised: Learning (EBCLR) which leverages the power of generative learning [8, 9, 10] by combining contrastive learning with energy-based models (EBMs).

Original: EBCLR complements the contrastive learning loss with a generative loss and it can be interpreted as learning the joint distribution of positive pairs.
Revised: EBCLR complements the contrastive learning loss with a generative loss and it can be interpreted as learning the joint distribution of positive pairs.

Original: In fact we demonstrate that the existing contrastive loss is a special case of the EBCLR loss if the generative term is not used.
Revised: In fact we demonstrate that the existing contrastive loss is a special case of the EBCLR loss if the generative term is not used.

Original: Although EBMs are notorious for being difficult to train due to their reliance on Stochastic Gradient Langevin Dynamics (SGLD) [11], another important contribution of this work is that we overcome this by appropriate modifications to SGLD.
Revised: Although EBMs are notorious for being difficult to train due to their reliance on Stochastic Gradient Langevin Dynamics (SGLD) [11], another important contribution of this work is that we overcome this by appropriate modifications to SGLD.

Original: Extensive experiments on a variety of small and medium-scale datasets demonstrate that EBCLR is robust to small numbers of negative pairs, and it outperforms SimCLR and MoCo v2 [12] in terms of sample efficiency and linear evaluation accuracy.
Revised: Extensive experiments on a variety of small and medium-scale datasets demonstrate that EBCLR is robust to small numbers of negative pairs, and it outperforms SimCLR and MoCo v2 [12] in terms of sample efficiency and linear evaluation accuracy.

Original: Hence, EBCLR opens up a new research direction for alleviating the dependence of contrastive methods on large datasets and large batches.
Revised: Hence, EBCLR opens up a new research direction for alleviating the dependence of contrastive methods on large datasets and large batches.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We propose a novel contrastive learning method called EBCLR which learns the joint distribution of positive pairs.
Revised: • We propose a novel contrastive learning method called EBCLR which learns the joint distribution of positive pairs.

Original: We show that EBCLR loss is equivalent to a combination of a contrastive term and a generative term (Section 3).
Revised: We show that EBCLR loss is equivalent to a combination of a contrastive term and a generative term (Section 3).

Original: To the best of our knowledge, this is the first work to apply EBMs to contrastive learning of visual representations.
Revised: To the best of our knowledge, this is the first work to apply EBMs to contrastive learning of visual representations.

Original: • We show that EBCLR offers two advantages over conventional contrastive learning methods: EBCLR is several times more sample efficient (Section 4.1) and robust to small batch sizes (Section 4.2).
Revised: • We show that EBCLR offers two advantages over conventional contrastive learning methods: EBCLR is several times more sample efficient (Section 4.1) and robust to small batch sizes (Section 4.2).

Original: These factors lead to a non-trivial performance gain for EBCLR.
Revised: These factors lead to a non-trivial performance gain for EBCLR.

Original: • We perform thorough ablation studies of the components of EBCLR: effect of changing the weight of the generative term (Section 4.3), effect of projection space dimension (Section 4.3), and the effect of the proposed SGLD modifications (Section 4.4).
Revised: • We perform thorough ablation studies of the components of EBCLR: effect of changing the weight of the generative term (Section 4.3), effect of projection space dimension (Section 4.3), and the effect of the proposed SGLD modifications (Section 4.4).

Original:  Related Works
Revised:  Related Works

Original: In this section, we go over related works necessary for understanding EBCLR.
Revised: In this section, we go over related works necessary for understanding EBCLR.

Original: In Appendix A, we give a more extensive review of relevant works for those not familiar with EBMs, contrastive learning, or generative models.
Revised: In Appendix A, we give a more extensive review of relevant works for those not familiar with EBMs, contrastive learning, or generative models.

Original: 2.1 Contrastive Learning
Revised: 2.1 Contrastive Learning

Original: For a given batch of images { x n } Nn =1 and two image transformations t , t ′ , contrastive learning methods first create two views v n = t ( x n ) , v ′ n = t ′ ( x n ) of each instance x n .
Revised: For a given batch of images { x n } Nn =1 and two image transformations t , t ′ , contrastive learning methods first create two views v n = t ( x n ) , v ′ n = t ′ ( x n ) of each instance x n .

Original: Here, the pair ( v n , v ′ m ) is called a positive pair if n = m and a negative pair if n ̸ = m .
Revised: Here, the pair ( v n , v ′ m ) is called a positive pair if n = m and a negative pair if n ̸ = m .

Original: Given a DNN f θ , the views are then embedded into the projection space by passing the views through f θ and normalizing.
Revised: Given a DNN f θ , the views are then embedded into the projection space by passing the views through f θ and normalizing.

Original: Contrastive methods train f θ to increase agreement between projections of positive pairs and decrease agreement between projections of negative pairs.
Revised: Contrastive methods train f θ to increase agreement between projections of positive pairs and decrease agreement between projections of negative pairs.

Original: Specifically, f θ is trained to maximize the InfoNCE objective [1].
Revised: Specifically, f θ is trained to maximize the InfoNCE objective [1].

Original: After training, outputs from the final layer or an intermediate layer of f θ are used for downstream tasks.
Revised: After training, outputs from the final layer or an intermediate layer of f θ are used for downstream tasks.

Original: There are numerous variants of contrastive methods.
Revised: There are numerous variants of contrastive methods.

Original: For instance, SimCLR [4] uses a composition of random cropping, random flipping, color jittering, color dropping, and blurring as the image transformation.
Revised: For instance, SimCLR [4] uses a composition of random cropping, random flipping, color jittering, color dropping, and blurring as the image transformation.

Original: Negative pairs are created by transforming different images within a batch.
Revised: Negative pairs are created by transforming different images within a batch.

Original: On the other hand, MoCo [12] maintains a queue of negative samples, so negative samples are not limited to views of images from the same batch.
Revised: On the other hand, MoCo [12] maintains a queue of negative samples, so negative samples are not limited to views of images from the same batch.

Original: 2.2 Energy-Based Models
Revised: 2.2 Energy-Based Models

Original: Given a scalar-valued energy function E θ ( v ) with parameter θ , an energy-based model (EBM) [13] defines a distribution by the formula
Revised: Given a scalar-valued energy function E θ ( v ) with parameter θ , an energy-based model (EBM) [13] defines a distribution by the formula

Original: [Equation]
Revised: [Equation]

Original: where Z ( θ ) is the partition function which guarantees q θ integrates to 1 .
Revised: where Z ( θ ) is the partition function which guarantees q θ integrates to 1 .

Original: Since there are essentially no restrictions on the choice of the energy function, EBMs have great flexibility in modeling distributions.
Revised: Since there are essentially no restrictions on the choice of the energy function, EBMs have great flexibility in modeling distributions.

Original: Hence, EBMs have been applied to a wide variety of machine learning tasks, such as dimensionality reduction via autoencoding [14], learning generative classifiers [8, 9, 10, 15], generating images [16], and training regression models [17, 18].
Revised: Hence, EBMs have been applied to a wide variety of machine learning tasks, such as dimensionality reduction via autoencoding [14], learning generative classifiers [8, 9, 10, 15], generating images [16], and training regression models [17, 18].

Original: Wang et al.
Revised: Wang et al.

Original: [19] have explored connections between EBMs and InfoNCE to enhance generative performance of EBMs.
Revised: [19] have explored connections between EBMs and InfoNCE to enhance generative performance of EBMs.

Original: However, to the best of our knowledge, this paper is the first to combine EBMs with contrastive learning for representation learning.
Revised: However, to the best of our knowledge, this paper is the first to combine EBMs with contrastive learning for representation learning.

Original: Given a target distribution, an EBM can be used to estimate its density p when we can only sample from p .
Revised: Given a target distribution, an EBM can be used to estimate its density p when we can only sample from p .

Original: One way of achieving this is by minimizing the Kullback-Leibler (KL) divergence between q θ and p that maximizes the expected log-likelihood of q θ under p [20]:
Revised: One way of achieving this is by minimizing the Kullback-Leibler (KL) divergence between q θ and p that maximizes the expected log-likelihood of q θ under p [20]:

Original: [Equation]
Revised: [Equation]

Original: Stochastic gradient ascent can be used to solve (2) [20].
Revised: Stochastic gradient ascent can be used to solve (2) [20].

Original: Specifically, the gradient of the expected log-likelihood with respect to the parameters θ can be shown to be
Revised: Specifically, the gradient of the expected log-likelihood with respect to the parameters θ can be shown to be

Original: [Equation]
Revised: [Equation]

Original: Hence, updating θ with (3) amounts to pushing up on the energy for samples from q θ and pushing down on the energy for samples from p .
Revised: Hence, updating θ with (3) amounts to pushing up on the energy for samples from q θ and pushing down on the energy for samples from p .

Original: This optimization method is also known as contrastive divergence [21].
Revised: This optimization method is also known as contrastive divergence [21].

Original: While the second term in (3) can be easily calculated as we have access to samples from p , the first term requires sampling from q θ .
Revised: While the second term in (3) can be easily calculated as we have access to samples from p , the first term requires sampling from q θ .

Original: Previous works [16, 22, 10, 15] have used Stochastic Gradient
Revised: Previous works [16, 22, 10, 15] have used Stochastic Gradient

Original: Langevin Dynamics (SGLD)
Revised: Langevin Dynamics (SGLD)

Original: [11] to generate samples from q θ .
Revised: [11] to generate samples from q θ .

Original: Specifically, given a sample v 0 from some proposal distribution q 0 , the iteration
Revised: Specifically, given a sample v 0 from some proposal distribution q 0 , the iteration

Original: [Equation]
Revised: [Equation]

Original: guarantees that the sequence { v t } converges to a sample from q θ assuming { α t } decays at a polynomial rate [11].
Revised: guarantees that the sequence { v t } converges to a sample from q θ assuming { α t } decays at a polynomial rate [11].

Original: However, SGLD requires an infinite number of steps until samples from the proposal distribution converge to samples from the target distribution.
Revised: However, SGLD requires an infinite number of steps until samples from the proposal distribution converge to samples from the target distribution.

Original: This is unfeasible, so in practice, only a finite number of steps along with constant step size, i.e. α t = α and constant noise variance σ t = σ 2 are used [16, 22, 10, 15].
Revised: This is unfeasible, so in practice, only a finite number of steps along with constant step size, i.e. α t = α and constant noise variance σ t = σ 2 are used [16, 22, 10, 15].

Original: Moreover, Yang and Ji [15] noted SGLD often generates samples with extreme pixel values that cause EBMs to diverge during training.
Revised: Moreover, Yang and Ji [15] noted SGLD often generates samples with extreme pixel values that cause EBMs to diverge during training.

Original: Hence, they have proposed proximal SGLD which clamps gradient values into an interval [ − δ, δ ] for a threshold δ > 0 .
Revised: Hence, they have proposed proximal SGLD which clamps gradient values into an interval [ − δ, δ ] for a threshold δ > 0 .

Original: Then, the update equation becomes
Revised: Then, the update equation becomes

Original: [Equation]
Revised: [Equation]

Original: for t = 0 , . . .
Revised: for t = 0 , . . .

Original: , T − 1 , where ϵ ∼ N (0 , σ 2 ) and clamp {· , δ } clamps each element of the input vector into [ − δ, δ ] .
Revised: , T − 1 , where ϵ ∼ N (0 , σ 2 ) and clamp {· , δ } clamps each element of the input vector into [ − δ, δ ] .

Original: In our work, we introduce additional modifications to SGLD which accelerate the convergence of EBCLR.
Revised: In our work, we introduce additional modifications to SGLD which accelerate the convergence of EBCLR.

Original: 𝑡 ∼ 𝒯 𝑣 𝑛 ′ 𝑣 𝑛 𝑧 𝑛′ ෤𝑣 𝑛 𝑧 𝑛 ǁ𝑧 𝑛
Revised: 𝑡 ∼ 𝒯 𝑣 𝑛 ′ 𝑣 𝑛 𝑧 𝑛′ ෤𝑣 𝑛 𝑧 𝑛 ǁ𝑧 𝑛

Original: [Figure]
Revised: [Figure]

Original: Loss 𝑥
Revised: Loss 𝑥

Original: 𝑛 −𝜆 ⋅𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ }) 𝜆 ⋅𝐸 𝜃 (𝑧 𝑛 ;{𝑧 𝑚 ′ })
Revised: 𝑛 −𝜆 ⋅𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ }) 𝜆 ⋅𝐸 𝜃 (𝑧 𝑛 ;{𝑧 𝑚 ′ })

Original: Replay Buffer
Revised: Replay Buffer

Original: Update
Revised: Update

Original: Shared Parameters 𝑓 𝜃
Revised: Shared Parameters 𝑓 𝜃

Original: 𝑓
Revised: 𝑓

Original: 𝜃 𝑓 𝜃
Revised: 𝜃 𝑓 𝜃

Original: Update ෤𝑣 𝑛
Revised: Update ෤𝑣 𝑛

Original: [Figure]
Revised: [Figure]

Original: SGLD
Revised: SGLD

Original: [Figure]
Revised: [Figure]

Original: ∇ ෤𝑣 𝑛 𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ })
Revised: ∇ ෤𝑣 𝑛 𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ })

Original: [Figure]
Revised: [Figure]

Original:  Theory 3.1 Energy-Based Contrastive Learning
Revised:  Theory 3.1 Energy-Based Contrastive Learning

Original: Let D be a distribution of images and T a distribution of stochastic image transformations.
Revised: Let D be a distribution of images and T a distribution of stochastic image transformations.

Original: Given x ∼ D and i.i.d.
Revised: Given x ∼ D and i.i.d.

Original: t, t ′ ∼ T , our goal is to approximate the joint distribution of the views p ( v, v ′ ) , where v = t ( x ) , v ′ =
Revised: t, t ′ ∼ T , our goal is to approximate the joint distribution of the views p ( v, v ′ ) , where v = t ( x ) , v ′ =

Original: t ′ ( x ) using the model distribution
Revised: t ′ ( x ) using the model distribution

Original: [Equation]
Revised: [Equation]

Original: 6) where Z ( θ ) is a normalization constant, τ > 0 is a temperature hyper-parameter, and z and z ′ are projections computed by passing the views v and v ′ through the DNN f θ and then normalizing to have unit norm.
Revised: 6) where Z ( θ ) is a normalization constant, τ > 0 is a temperature hyper-parameter, and z and z ′ are projections computed by passing the views v and v ′ through the DNN f θ and then normalizing to have unit norm.

Original: We now explain the intuitive meaning of matching q θ to p .
Revised: We now explain the intuitive meaning of matching q θ to p .

Original: Our key idea is to use p ( v, v ′ ) as a measure of semantic similarity of v and v ′ .
Revised: Our key idea is to use p ( v, v ′ ) as a measure of semantic similarity of v and v ′ .

Original: If two images v and v ′ are semantically similar, they are likely to be transformations of similar images.
Revised: If two images v and v ′ are semantically similar, they are likely to be transformations of similar images.

Original: So, p ( v, v ′ ) will be high when v and v ′ are semantically similar and low otherwise.
Revised: So, p ( v, v ′ ) will be high when v and v ′ are semantically similar and low otherwise.

Original: Suppose q θ successfully approximates p .
Revised: Suppose q θ successfully approximates p .

Original: If we equate p ( v, v ′ ) to q θ ( v, v ′ ) in (6) and solve for ∥ z − z ′ ∥ , we see that the distance between z and z ′ will become a monotone increasing function of 1 /p ( v, v ′ ) , which is the inverse of semantic similarity of v and v ′ .
Revised: If we equate p ( v, v ′ ) to q θ ( v, v ′ ) in (6) and solve for ∥ z − z ′ ∥ , we see that the distance between z and z ′ will become a monotone increasing function of 1 /p ( v, v ′ ) , which is the inverse of semantic similarity of v and v ′ .

Original: So, semantically similar images will have nearby projections, and dissimilar images will have distant projections.
Revised: So, semantically similar images will have nearby projections, and dissimilar images will have distant projections.

Original: This idea is illustrated in Figure 1.
Revised: This idea is illustrated in Figure 1.

Original: To approximate p using q θ , we train f θ to maximize the expected log-likelihood of q θ under p :
Revised: To approximate p using q θ , we train f θ to maximize the expected log-likelihood of q θ under p :

Original: [Equation]
Revised: [Equation]

Original: In order to solve this problem with stochastic gradient ascent, we could naively extend (3) to the setting of joint distributions to obtain the following result.
Revised: In order to solve this problem with stochastic gradient ascent, we could naively extend (3) to the setting of joint distributions to obtain the following result.

Original: Proposition 1.
Revised: Proposition 1.

Original: The the joint distribution (6) can be formulated as an EBM
Revised: The the joint distribution (6) can be formulated as an EBM

Original: [Equation]
Revised: [Equation]

Original: and the gradient of the objective of (7) is given by
Revised: and the gradient of the objective of (7) is given by

Original: [Equation]
Revised: [Equation]

Original: However, computing the first expectation in (9) requires sampling pairs of views ( v, v ′ ) from q θ ( v, v ′ ) via SGLD, which could be expensive.
Revised: However, computing the first expectation in (9) requires sampling pairs of views ( v, v ′ ) from q θ ( v, v ′ ) via SGLD, which could be expensive.

Original: To avert this problem, we use Bayes’ rule to decompose
Revised: To avert this problem, we use Bayes’ rule to decompose

Original: E p [log q θ ( v, v ′ )]
Revised: E p [log q θ ( v, v ′ )]

Original: =
Revised: =

Original: E p [log q θ ( v ′ | v )]
Revised: E p [log q θ ( v ′ | v )]

Original: + E p [log q θ ( v )] where q θ ( v ) = (cid:90) q θ ( v, v ′ ) dv ′ .
Revised: + E p [log q θ ( v )] where q θ ( v ) = (cid:90) q θ ( v, v ′ ) dv ′ .

Original: In the first equation of (10), the first and second terms at the RHS will be referred to as discriminative and generative terms, respectively, throughout the paper.
Revised: In the first equation of (10), the first and second terms at the RHS will be referred to as discriminative and generative terms, respectively, throughout the paper.

Original: A similar decomposition was used by
Revised: A similar decomposition was used by

Original: Grathwohl et al.
Revised: Grathwohl et al.

Original: [10] in the setting of learning generative classifiers.
Revised: [10] in the setting of learning generative classifiers.

Original: Furthermore, we add a hyper-parameter λ to balance the strength of the discriminative term and the generative term.
Revised: Furthermore, we add a hyper-parameter λ to balance the strength of the discriminative term and the generative term.

Original: The advantage of this modification will be discussed in Section 4.3.
Revised: The advantage of this modification will be discussed in Section 4.3.

Original: This yields our
Revised: This yields our

Original: Energy-Based Contrastive Learning (EBCLR) objective
Revised: Energy-Based Contrastive Learning (EBCLR) objective

Original: [Equation]
Revised: [Equation]

Original: The discriminative term can be easily differentiated since the partition function Z ( θ ) cancels out when q θ ( v, v ′ ) is divided by q θ ( v ) .
Revised: The discriminative term can be easily differentiated since the partition function Z ( θ ) cancels out when q θ ( v, v ′ ) is divided by q θ ( v ) .

Original: However, the generative term still contains Z ( θ ) .
Revised: However, the generative term still contains Z ( θ ) .

Original: We now present our key result, which is used to maximize (11).
Revised: We now present our key result, which is used to maximize (11).

Original: The proof is deferred to Appendix C.1.
Revised: The proof is deferred to Appendix C.1.

Original: Theorem 2.
Revised: Theorem 2.

Original: The marginal distribution in (10) can be formulated as an EBM
Revised: The marginal distribution in (10) can be formulated as an EBM

Original: [Equation]
Revised: [Equation]

Original: where Z ( θ ) is the partition function in (6) , and the gradient of the generative term is given by
Revised: where Z ( θ ) is the partition function in (6) , and the gradient of the generative term is given by

Original: [Equation]
Revised: [Equation]

Original: Thus, the gradient of the EBCLR objective is
Revised: Thus, the gradient of the EBCLR objective is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 suggests that the EBM for the joint distribution can be learned by computing the gradients of the discriminative term and the EBM for the marginal distribution.
Revised: Theorem 2 suggests that the EBM for the joint distribution can be learned by computing the gradients of the discriminative term and the EBM for the marginal distribution.

Original: Moreover, we only need to sample v from q θ ( v ) to compute the second expectation in (14).
Revised: Moreover, we only need to sample v from q θ ( v ) to compute the second expectation in (14).

Original: 3.2 Approximating the EBCLR Objective
Revised: 3.2 Approximating the EBCLR Objective

Original: To implement EBCLR, we need to approximate expectations in (11) with their empirical means.
Revised: To implement EBCLR, we need to approximate expectations in (11) with their empirical means.

Original: Suppose samples { ( v n , v ′ n ) }
Revised: Suppose samples { ( v n , v ′ n ) }

Original: Nn =1 from p ( v, v ′ ) are given, and let { ( z n , z ′ n ) }
Revised: Nn =1 from p ( v, v ′ ) are given, and let { ( z n , z ′ n ) }

Original: Nn =1 be the corresponding projections.
Revised: Nn =1 be the corresponding projections.

Original: As the learning goal is to make q θ ( v n , v ′ n ) approximate the joint probability density function p ( v n , v ′ n ) , the empirical mean (cid:98) q θ ( v n ) can be defined as:
Revised: As the learning goal is to make q θ ( v n , v ′ n ) approximate the joint probability density function p ( v n , v ′ n ) , the empirical mean (cid:98) q θ ( v n ) can be defined as:

Original: [Equation]
Revised: [Equation]

Original: where the sum is over the collection of v ′ m defined as
Revised: where the sum is over the collection of v ′ m defined as

Original: [Equation]
Revised: [Equation]

Original: and N ′ :
Revised: and N ′ :

Original: = |{ v ′ m : v ′ m ̸
Revised: = |{ v ′ m : v ′ m ̸

Original: = v n }| = 2 N − 1 .
Revised: = v n }| = 2 N − 1 .

Original: One could also use a simpler form of the empirical mean:
Revised: One could also use a simpler form of the empirical mean:

Original: [Equation]
Revised: [Equation]

Original: Similarly, q θ ( v ′ | v ) in (11), which should approximate the conditional probability density p ( v ′ | v ) , can be represented in terms of q θ ( v n , v ′ n ) .
Revised: Similarly, q θ ( v ′ | v ) in (11), which should approximate the conditional probability density p ( v ′ | v ) , can be represented in terms of q θ ( v n , v ′ n ) .

Original: Specifically, we have
Revised: Specifically, we have

Original: [Equation]
Revised: [Equation]

Original: It is then immediately apparent that the empirical form of the discriminative term using (18) is a particular instance of the contrastive learning objective such as InfoNCE and SimCLR.
Revised: It is then immediately apparent that the empirical form of the discriminative term using (18) is a particular instance of the contrastive learning objective such as InfoNCE and SimCLR.

Original: Hence,
Revised: Hence,

Original: EBCLR can be interpreted as complementing contrastive learning with a generative term defined by an EBM.
Revised: EBCLR can be interpreted as complementing contrastive learning with a generative term defined by an EBM.

Original: We will demonstrate in Section 4.1 that the generative term offers significant advantages over other contrastive learning methods.
Revised: We will demonstrate in Section 4.1 that the generative term offers significant advantages over other contrastive learning methods.

Original: For the second term, we use the simpler form of the empirical mean in (17):
Revised: For the second term, we use the simpler form of the empirical mean in (17):

Original: [Equation]
Revised: [Equation]

Original: We could also use (15) as the empirical mean, but either choice showed identical performance (see
Revised: We could also use (15) as the empirical mean, but either choice showed identical performance (see

Original: Appendix E.3).
Revised: Appendix E.3).

Original: So, we have found (15) to be not worth the additional complexity, and have resorted to the simpler approximation (17) instead.
Revised: So, we have found (15) to be not worth the additional complexity, and have resorted to the simpler approximation (17) instead.

Original: In Appendix C.2, we theoretically justify that EBCLR will work as intended even with the approximations (15) or (17).
Revised: In Appendix C.2, we theoretically justify that EBCLR will work as intended even with the approximations (15) or (17).

Original: If we compare (19) with (12), we can see that this approximation of q θ ( v ) yields the energy function (after ignoring the constant log N )
Revised: If we compare (19) with (12), we can see that this approximation of q θ ( v ) yields the energy function (after ignoring the constant log N )

Original: [Equation]
Revised: [Equation]

Original: 3.3 Modifications to SGLD
Revised: 3.3 Modifications to SGLD

Original: According to Theorem 2, we need samples from the marginal q θ ( v ) to calculate the second expectation in (14).
Revised: According to Theorem 2, we need samples from the marginal q θ ( v ) to calculate the second expectation in (14).

Original: Hence, we apply proximal SGLD (5) with the energy function (20) to sample from q θ ( v ) as
Revised: Hence, we apply proximal SGLD (5) with the energy function (20) to sample from q θ ( v ) as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for t = 0 , . . .
Revised: for t = 0 , . . .

Original: , T − 1 , where ϵ ∼ N (0 , σ 2 ) .
Revised: , T − 1 , where ϵ ∼ N (0 , σ 2 ) .

Original: We make three additional modifications to proximal
Revised: We make three additional modifications to proximal

Original: SGLD to expedite the training process.
Revised: SGLD to expedite the training process.

Original: From here on, we will be referring to proximal SGLD in (5) when we say SGLD.
Revised: From here on, we will be referring to proximal SGLD in (5) when we say SGLD.

Original: First, we initialize SGLD from generated samples from previous iterations, and with probability ρ , we reinitialize SGLD chains from samples from a proposal distribution q 0 .
Revised: First, we initialize SGLD from generated samples from previous iterations, and with probability ρ , we reinitialize SGLD chains from samples from a proposal distribution q 0 .

Original: This is achieved by keeping a replay buffer B of SGLD samples from previous iterations.
Revised: This is achieved by keeping a replay buffer B of SGLD samples from previous iterations.

Original: This technique of maintaining a replay buffer has also been used in previous works and has proven to be crucial for stabilizing and accelerating the convergence of EBMs [16, 10, 15].
Revised: This technique of maintaining a replay buffer has also been used in previous works and has proven to be crucial for stabilizing and accelerating the convergence of EBMs [16, 10, 15].

Original: Second, the proposal distribution q 0 is set to be the data distribution p ( v ) .
Revised: Second, the proposal distribution q 0 is set to be the data distribution p ( v ) .

Original: This choice differs from those of previous works [16, 10, 15] which have either used the uniform distribution or a mixture of
Revised: This choice differs from those of previous works [16, 10, 15] which have either used the uniform distribution or a mixture of

Original: Gaussians as the proposal distribution.
Revised: Gaussians as the proposal distribution.

Original: Finally, we use multi-stage SGLD (MSGLD) , which adaptively controls the magnitude of noise added in SGLD.
Revised: Finally, we use multi-stage SGLD (MSGLD) , which adaptively controls the magnitude of noise added in SGLD.

Original: For each sample ˜ v in the replay buffer B , we keep a count κ ˜ v of number of times it has been used as the initial point of SGLD.
Revised: For each sample ˜ v in the replay buffer B , we keep a count κ ˜ v of number of times it has been used as the initial point of SGLD.

Original: For samples with a low count, we use noise of high variance, and for samples with a high count, we use noise of low variance.
Revised: For samples with a low count, we use noise of high variance, and for samples with a high count, we use noise of low variance.

Original: Specifically, in (5), we set
Revised: Specifically, in (5), we set

Original: [Equation]
Revised: [Equation]

Original: where [ · ] + := max { 0 , ·} , σ 2max and σ 2min are the upper and lower bounds on the noise variance, respectively, and K controls the decay rate of noise variance.
Revised: where [ · ] + := max { 0 , ·} , σ 2max and σ 2min are the upper and lower bounds on the noise variance, respectively, and K controls the decay rate of noise variance.

Original: The purpose of this technique is to facilitate quick exploration of the modes of q θ and still guarantee SGLD generates samples with sufficiently low energy.
Revised: The purpose of this technique is to facilitate quick exploration of the modes of q θ and still guarantee SGLD generates samples with sufficiently low energy.

Original: The pseudocodes for MSGLD and EBCLR are given in Algorithms 1 and 2, respectively, in Appendix B, and the overall learning flow of EBCLR is described in Figure 2.
Revised: The pseudocodes for MSGLD and EBCLR are given in Algorithms 1 and 2, respectively, in Appendix B, and the overall learning flow of EBCLR is described in Figure 2.

Original:  Experiments
Revised:  Experiments

Original: We now describe the experimental settings.
Revised: We now describe the experimental settings.

Original: A complete description is deferred to Appendix D.
Revised: A complete description is deferred to Appendix D.

Original: Baseline methods and datasets.
Revised: Baseline methods and datasets.

Original: The baseline methods are SimCLR, MoCo v2, SimSiam, and
Revised: The baseline methods are SimCLR, MoCo v2, SimSiam, and

Original: BYOL.
Revised: BYOL.

Original: The hyper-parameters are chosen closely following the original works [4, 12, 7, 6].
Revised: The hyper-parameters are chosen closely following the original works [4, 12, 7, 6].

Original: We use four datasets: MNIST [23], Fashion MNIST (FMNIST)
Revised: We use four datasets: MNIST [23], Fashion MNIST (FMNIST)

Original: [24], CIFAR10, and CIFAR100 [25].
Revised: [24], CIFAR10, and CIFAR100 [25].

Original: DNN architecture.
Revised: DNN architecture.

Original: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is the projection network.
Revised: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is the projection network.

Original: Rather than using the output of f θ for downstream tasks, we follow previous works [4, 5, 1, 2, 3, 6, 7] and use the output of ϕ θ instead.
Revised: Rather than using the output of f θ for downstream tasks, we follow previous works [4, 5, 1, 2, 3, 6, 7] and use the output of ϕ θ instead.

Original: In our experiments, we set ϕ θ to be a ResNet-18 [26] up to the global average pooling layer and π θ to be a 2-layer MLP with output dimension 128 .
Revised: In our experiments, we set ϕ θ to be a ResNet-18 [26] up to the global average pooling layer and π θ to be a 2-layer MLP with output dimension 128 .

Original: However, we remove batch normalization because batch normalization hurts SGLD [16].
Revised: However, we remove batch normalization because batch normalization hurts SGLD [16].

Original: We also replace ReLU with leaky ReLU to expedite the convergence of SGLD.
Revised: We also replace ReLU with leaky ReLU to expedite the convergence of SGLD.

Original: For the baselines, we use settings proposed in the original works while keeping the backbone fixed to be ResNet-18.
Revised: For the baselines, we use settings proposed in the original works while keeping the backbone fixed to be ResNet-18.

Original: Evaluation.
Revised: Evaluation.

Original: We evaluate the representations by training a linear classifier on top of frozen ϕ θ .
Revised: We evaluate the representations by training a linear classifier on top of frozen ϕ θ .

Original: [Table]
Revised: [Table]

Original: 4.1 Comparison with Baselines
Revised: 4.1 Comparison with Baselines

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: We use batch size 128 for EBCLR and batch size 256 for the baseline methods following Wang et.
Revised: We use batch size 128 for EBCLR and batch size 256 for the baseline methods following Wang et.

Original: al [27] and train each method for 100 epochs.
Revised: al [27] and train each method for 100 epochs.

Original: Table 1 shows the result of training each method forepochs.
Revised: Table 1 shows the result of training each method forepochs.

Original: Observe that EBCLR consistently outperforms all baseline methods in terms of linear evaluation accuracy.
Revised: Observe that EBCLR consistently outperforms all baseline methods in terms of linear evaluation accuracy.

Original: Moreover, relative efficiency indicates EBCLR is capable of achieving the same level of performance as the baseline methods with much fewer training epochs.
Revised: Moreover, relative efficiency indicates EBCLR is capable of achieving the same level of performance as the baseline methods with much fewer training epochs.

Original: Concretely, we observe at least × 4 acceleration in terms of epochs compared to contrastive methods.
Revised: Concretely, we observe at least × 4 acceleration in terms of epochs compared to contrastive methods.

Original: Hence, EBCLR is a much more desirable choice than SimCLR or MoCo v2 for learning visual representations when we have a small number of training samples.
Revised: Hence, EBCLR is a much more desirable choice than SimCLR or MoCo v2 for learning visual representations when we have a small number of training samples.

Original: We also investigate the transfer learning performance of EBCLR.
Revised: We also investigate the transfer learning performance of EBCLR.

Original: Table 2 compares the transfer learning accuracies.
Revised: Table 2 compares the transfer learning accuracies.

Original: EBCLR always outperforms the baseline methods, and the performance gap is especially large on CIFAR10 and CIFAR100.
Revised: EBCLR always outperforms the baseline methods, and the performance gap is especially large on CIFAR10 and CIFAR100.

Original: This indicates EBCLR learns visual representations that generalize well across datasets.
Revised: This indicates EBCLR learns visual representations that generalize well across datasets.

Original: Repeating the above experiments with longer training or KNN classification led to similar conclusions (see Appendixes E.1 and E.4, respectively).
Revised: Repeating the above experiments with longer training or KNN classification led to similar conclusions (see Appendixes E.1 and E.4, respectively).

Original: 4.2 Effect of Reducing Negative Pairs
Revised: 4.2 Effect of Reducing Negative Pairs

Original: We compared the performances of EBCLR and SimCLR as we reduced the number of negative pairs per positive pair.
Revised: We compared the performances of EBCLR and SimCLR as we reduced the number of negative pairs per positive pair.

Original: For MoCo v2, the negative samples are provided by a queue updated by a momentum encoder.
Revised: For MoCo v2, the negative samples are provided by a queue updated by a momentum encoder.

Original: On the other hand, for EBCLR and SimCLR, negative samples come from the same batch as the positive pair.
Revised: On the other hand, for EBCLR and SimCLR, negative samples come from the same batch as the positive pair.

Original: So, we did not have a way of fairly comparing EBCLR and SimCLR with MoCo v2.
Revised: So, we did not have a way of fairly comparing EBCLR and SimCLR with MoCo v2.

Original: Hence, we excluded MoCo v2 from this experiment.
Revised: Hence, we excluded MoCo v2 from this experiment.

Original: We note that, according to (18), given a batch of size N , we obtain 2 N − 2 negative pairs for each positive pair.
Revised: We note that, according to (18), given a batch of size N , we obtain 2 N − 2 negative pairs for each positive pair.

Original: SimCLR also has 2 N − 2 negative pairs for each positive pair.
Revised: SimCLR also has 2 N − 2 negative pairs for each positive pair.

Original: Hence, we can conveniently compare the sensitivity of EBCLR and SimCLR to the number of negative pairs by varying the batch size.
Revised: Hence, we can conveniently compare the sensitivity of EBCLR and SimCLR to the number of negative pairs by varying the batch size.

Original: Table 3 shows the result of training each method for 100 epochs with batch sizes in { 16 , 64 , 128 } .
Revised: Table 3 shows the result of training each method for 100 epochs with batch sizes in { 16 , 64 , 128 } .

Original: We make three important observations.
Revised: We make three important observations.

Original: First, EBCLR consistently beats SimCLR in terms of linear evaluation accuracy for every batch size.
Revised: First, EBCLR consistently beats SimCLR in terms of linear evaluation accuracy for every batch size.

Original: Second, EBCLR is invariant to the choice of batch size.
Revised: Second, EBCLR is invariant to the choice of batch size.

Original: This contrasts with SimCLR whose performance degrades as batch size decreases.
Revised: This contrasts with SimCLR whose performance degrades as batch size decreases.

Original: Consequently,
Revised: Consequently,

Original: EBCLR with batch size 16 beats SimCLR with batch size 128 .
Revised: EBCLR with batch size 16 beats SimCLR with batch size 128 .

Original: Finally, as a byproduct of the second observation, the efficiency of EBCLR relative to SimCLR increases as batch size decreases.
Revised: Finally, as a byproduct of the second observation, the efficiency of EBCLR relative to SimCLR increases as batch size decreases.

Original: These properties make EBCLR suitable for situations where we cannot use large batch sizes, e.g., when we
Revised: These properties make EBCLR suitable for situations where we cannot use large batch sizes, e.g., when we

Original: [Table]
Revised: [Table]

Original: have a small number of GPUs.
Revised: have a small number of GPUs.

Original: Repeating the experiments with longer training or KNN classification again led to similar conclusions (see Appendixes E.2 and E.4, respectively).
Revised: Repeating the experiments with longer training or KNN classification again led to similar conclusions (see Appendixes E.2 and E.4, respectively).

Original: 4.3 Effect of λ and Projection Dimension
Revised: 4.3 Effect of λ and Projection Dimension

Original: [Figure]
Revised: [Figure]

Original: We explored the effect of changing the hyperparameter λ which controls the importance of the generative term relative to the discriminative term (see Equation (11)).
Revised: We explored the effect of changing the hyperparameter λ which controls the importance of the generative term relative to the discriminative term (see Equation (11)).

Original: Figure 3a shows the performance of EBCLR with various values of λ as training progresses.
Revised: Figure 3a shows the performance of EBCLR with various values of λ as training progresses.

Original: We observe that naively using λ = 1 . 0 leads to poor results.
Revised: We observe that naively using λ = 1 . 0 leads to poor results.

Original: The performance peaks at λ = 0 .
Revised: The performance peaks at λ = 0 .

Original: 1 , and then degrades as we further decrease λ .
Revised: 1 , and then degrades as we further decrease λ .

Original: This result has two crucial implications.
Revised: This result has two crucial implications.

Original: First, the generative term plays a non-trivial role in
Revised: First, the generative term plays a non-trivial role in

Original: EBCLR.
Revised: EBCLR.

Original: Second, we need to strike a right balance between the discriminative term and the generative term to achieve good performance on downstream tasks 1 .
Revised: Second, we need to strike a right balance between the discriminative term and the generative term to achieve good performance on downstream tasks 1 .

Original: We also investigated the effect of varying the output dimension of π θ .
Revised: We also investigated the effect of varying the output dimension of π θ .

Original: Figure 3b shows linear evaluation results for projection dimensions in { 128 , 256 , 512 } .
Revised: Figure 3b shows linear evaluation results for projection dimensions in { 128 , 256 , 512 } .

Original: We observe that the projection dimension has essentially no influence on the training process.
Revised: We observe that the projection dimension has essentially no influence on the training process.

Original: In this respect, EBCLR resembles
Revised: In this respect, EBCLR resembles

Original: SimCLR which is also invariant to the output dimension (see Figure 8 in the work by Chen et.
Revised: SimCLR which is also invariant to the output dimension (see Figure 8 in the work by Chen et.

Original: al [4]).
Revised: al [4]).

Original: 4.4 Effect of SGLD Modifications
Revised: 4.4 Effect of SGLD Modifications

Original: We now study the roles of the three SGLD modifications proposed in Section 3.3.
Revised: We now study the roles of the three SGLD modifications proposed in Section 3.3.

Original: Figure 4 shows the results of varying one parameter of MSGLD while keeping the others fixed.
Revised: Figure 4 shows the results of varying one parameter of MSGLD while keeping the others fixed.

Original: Effect of reintialization frequency ρ .
Revised: Effect of reintialization frequency ρ .

Original: Figure 4a displays linear evaluation results for ρ ∈ { 0 . 0 , 0 . 2 , 1 . 0 } .
Revised: Figure 4a displays linear evaluation results for ρ ∈ { 0 . 0 , 0 . 2 , 1 . 0 } .

Original: We note that setting ρ = 1 . 0 is equivalent to removing the replay buffer.
Revised: We note that setting ρ = 1 . 0 is equivalent to removing the replay buffer.

Original: Also, setting ρ = 0 . 0 is equivalent to never reinitializing SGLD chains.
Revised: Also, setting ρ = 0 . 0 is equivalent to never reinitializing SGLD chains.

Original: Initially, ρ = 0 . 0 shows the best performance, as SGLD quickly reaches samples of lower energy.
Revised: Initially, ρ = 0 . 0 shows the best performance, as SGLD quickly reaches samples of lower energy.

Original: However, learning then slows down because of the lack of diversity of samples in the replay buffer B .
Revised: However, learning then slows down because of the lack of diversity of samples in the replay buffer B .

Original: This implies that it is necessary to set ρ > 0 in order to learn good representations.
Revised: This implies that it is necessary to set ρ > 0 in order to learn good representations.

Original: On the other hand, ρ = 1 . 0 shows slow convergence in the beginning because samples in the replay buffer are not given enough iterations to reach low energy.
Revised: On the other hand, ρ = 1 . 0 shows slow convergence in the beginning because samples in the replay buffer are not given enough iterations to reach low energy.

Original: Although it does beat ρ = 0 . 0 at latter epochs, it still often performs worse than ρ = 0 .
Revised: Although it does beat ρ = 0 . 0 at latter epochs, it still often performs worse than ρ = 0 .

Original: Moreover, it is not sample-efficient compared to ρ = 0 .
Revised: Moreover, it is not sample-efficient compared to ρ = 0 .

Original: 2 since we have to provide an entire batch of new samples for reinitializing SGLD chains at each iteration.
Revised: 2 since we have to provide an entire batch of new samples for reinitializing SGLD chains at each iteration.

Original: Given the above observations, it is clear why the intermediate value 0 .
Revised: Given the above observations, it is clear why the intermediate value 0 .

Original: 2 is the best choice out of ρ ∈ { 0 . 0 , 0 . 2 , 1 . 0 } .
Revised: 2 is the best choice out of ρ ∈ { 0 . 0 , 0 . 2 , 1 . 0 } .

Original: ρ = 0 .
Revised: ρ = 0 .

Original: 2 allows enough time for samples in the replay buffer to reach low energy while still maintaining the diversity of samples in B .
Revised: 2 allows enough time for samples in the replay buffer to reach low energy while still maintaining the diversity of samples in B .

Original: Also, it is sample-efficient compared to ρ = 1 . 0 .
Revised: Also, it is sample-efficient compared to ρ = 1 . 0 .

Original: Effect of proposal distribution q 0 .
Revised: Effect of proposal distribution q 0 .

Original: Figure 4b compares linear evaluation accuracies with q 0 as the uniform distribution and q 0 = p ( v ) .
Revised: Figure 4b compares linear evaluation accuracies with q 0 as the uniform distribution and q 0 = p ( v ) .

Original: We observe prominent acceleration in the initial epochs for q 0 = p ( v ) .
Revised: We observe prominent acceleration in the initial epochs for q 0 = p ( v ) .

Original: Hence, we can conclude that this choice of proposal distribution is crucial for the high efficiency of EBCLR compared to the baseline methods in Tables 1 and 3.
Revised: Hence, we can conclude that this choice of proposal distribution is crucial for the high efficiency of EBCLR compared to the baseline methods in Tables 1 and 3.

Original: [Figure]
Revised: [Figure]

Original: We believe this acceleration effect can be explained by the work of Hinton [21].
Revised: We believe this acceleration effect can be explained by the work of Hinton [21].

Original: Specifically, let us observe that the EBM update equation (3) pushes up the energy on the model distribution q θ .
Revised: Specifically, let us observe that the EBM update equation (3) pushes up the energy on the model distribution q θ .

Original: In the implementation of EBCLR with q 0 = p ( v ) , however, q θ is replaced by the distribution of samples created by a finite number of (noisy) gradient steps on real data points (see Section 3.3).
Revised: In the implementation of EBCLR with q 0 = p ( v ) , however, q θ is replaced by the distribution of samples created by a finite number of (noisy) gradient steps on real data points (see Section 3.3).

Original: Hence, the modified
Revised: Hence, the modified

Original: EMB update equation contains the curvature information of the data manifold.
Revised: EMB update equation contains the curvature information of the data manifold.

Original: This curvature information may expedite the training process of EBCLR.
Revised: This curvature information may expedite the training process of EBCLR.

Original: For a detailed discussion on this, we refer the readers to
Revised: For a detailed discussion on this, we refer the readers to

Original: Section 3 of the work by Hinton [21].
Revised: Section 3 of the work by Hinton [21].

Original: Comparison of SGLD and MSGLD.
Revised: Comparison of SGLD and MSGLD.

Original: Figure 4c shows results with SGLD with σ ∈ { 0 . 01 , 0 . 05 } and
Revised: Figure 4c shows results with SGLD with σ ∈ { 0 . 01 , 0 . 05 } and

Original: MSGLD with σ min = 0 .
Revised: MSGLD with σ min = 0 .

Original: and σ max = 0 . 05 .
Revised: and σ max = 0 . 05 .

Original: We note that setting σ min = σ max reduces MSGLD
Revised: We note that setting σ min = σ max reduces MSGLD

Original: to
Revised: to

Original: SGLD.
Revised: SGLD.

Original: We observe σ = 0 .
Revised: We observe σ = 0 .

Original: 01 initially shows fast convergence but then saturates due to the lack of diversity of generated samples.
Revised: 01 initially shows fast convergence but then saturates due to the lack of diversity of generated samples.

Original: On the other hand, σ = 0 .initially has the worst performance but eventually beats σ = 0 .
Revised: On the other hand, σ = 0 .initially has the worst performance but eventually beats σ = 0 .

Original: 01 since σ = 0 .
Revised: 01 since σ = 0 .

Original: 05 quickly explores the modes of q θ .
Revised: 05 quickly explores the modes of q θ .

Original: MSGLD inherits the best of both settings.
Revised: MSGLD inherits the best of both settings.

Original: Specifically, MSGLD is as fast as σ = 0 .
Revised: Specifically, MSGLD is as fast as σ = 0 .

Original: 01 in the beginning, and it does not suffer from the saturation problem.
Revised: 01 in the beginning, and it does not suffer from the saturation problem.

Original:  Limitations and Societal Impacts
Revised:  Limitations and Societal Impacts

Original: Limitations.
Revised: Limitations.

Original: The main limitation of our work is of scale.
Revised: The main limitation of our work is of scale.

Original: While EBCLR demonstrates superior sample efficiency, it requires inner SGLD iterations (which cannot be parallelized) and a replay buffer
Revised: While EBCLR demonstrates superior sample efficiency, it requires inner SGLD iterations (which cannot be parallelized) and a replay buffer

Original: B .
Revised: B .

Original: These two components increase the computational burden of EBCLR.
Revised: These two components increase the computational burden of EBCLR.

Original: So, we found it difficult to apply EBCLR to large-scale data such as ImageNet.
Revised: So, we found it difficult to apply EBCLR to large-scale data such as ImageNet.

Original: However, we note that inner SGLD iterations and the replay buffer are not particular limitations of EBCLR, but limitations of EBMs in general.
Revised: However, we note that inner SGLD iterations and the replay buffer are not particular limitations of EBCLR, but limitations of EBMs in general.

Original: Given the increasing efforts to overcome these limitations such as Proximal-YOPO-SGLD (for more discussion, see Appendix F), we believe EBCLR will eventually be applicable to larger data.
Revised: Given the increasing efforts to overcome these limitations such as Proximal-YOPO-SGLD (for more discussion, see Appendix F), we believe EBCLR will eventually be applicable to larger data.

Original: Social Impacts.
Revised: Social Impacts.

Original: We generally expect positive outcomes from this research.
Revised: We generally expect positive outcomes from this research.

Original: Further development of EBCLR can mitigate the need for large amount of data and large batch sizes to learn good representations and ultimately lead to a reduction in resource consumption.
Revised: Further development of EBCLR can mitigate the need for large amount of data and large batch sizes to learn good representations and ultimately lead to a reduction in resource consumption.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we proposed EBCLR which combines contrastive learning with EBMs.
Revised: In this work, we proposed EBCLR which combines contrastive learning with EBMs.

Original: This amalgamation of ideas has led to both theoretical and practical contributions.
Revised: This amalgamation of ideas has led to both theoretical and practical contributions.

Original: Theoretically, EBCLR associates distance in the projection space with the density of positive samples.
Revised: Theoretically, EBCLR associates distance in the projection space with the density of positive samples.

Original: Since the distribution of positive samples reflects the semantic similarity of images, EBCLR is capable of learning good visual representations.
Revised: Since the distribution of positive samples reflects the semantic similarity of images, EBCLR is capable of learning good visual representations.

Original: Practically, EBCLR is several times more sample-efficient than conventional contrastive and non-contrastive learning approaches and is robust to small numbers of negative pairs.
Revised: Practically, EBCLR is several times more sample-efficient than conventional contrastive and non-contrastive learning approaches and is robust to small numbers of negative pairs.

Original: Hence, EBCLR is applicable even in scenarios with limited data or devices.
Revised: Hence, EBCLR is applicable even in scenarios with limited data or devices.

Original: We believe that EBCLR makes representation learning available to a wider range of machine learning practitioners.
Revised: We believe that EBCLR makes representation learning available to a wider range of machine learning practitioners.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work was supported by the National Research Foundation of Korea under Grant NRF2020R1A2B5B03001980, KAIST Key Research Institute (Interdisciplinary Research Group) Project, and Field-oriented Technology Development Project for Customs Administration through National
Revised: This work was supported by the National Research Foundation of Korea under Grant NRF2020R1A2B5B03001980, KAIST Key Research Institute (Interdisciplinary Research Group) Project, and Field-oriented Technology Development Project for Customs Administration through National

Original: Research Foundation of Korea(NRF) funded by the Ministry of Science & ICT and Korea Customs
Revised: Research Foundation of Korea(NRF) funded by the Ministry of Science & ICT and Korea Customs

Original: Service(**NRF-2021M3I1A1097938**).
Revised: Service(**NRF-2021M3I1A1097938**).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix C. 3.
Revised: [Yes] See Appendix C. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] Each EBM was trained once due to computation costs.
Revised: [No] Each EBM was trained once due to computation costs.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix D. 4.
Revised: [Yes] See Appendix D. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re us- ing/curating?
Revised: and how consent was obtained from people whose data you’re us- ing/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: A Additional Background A.1 Contrastive Divergence and Contrastive Learning

Original: Contrastive divergence, initially proposed by Hinton et al.

Original: [21], is a particular method for solving the maximum- likelihood estimation problem for EBMs.

Original: Intuitively, it proceeds by minimizing the energy function on data and maximizing the energy function on (possibly short-run) MCMC samples of the current EBM distribution through gradient ascent.

Original: In fact, we use an instance of contrastive divergence to optimize the generative term in our EBCLR objective (11).

Original: We also disambiguate contrastive learning from contrastive divergence.

Original: They are similar in the sense that they contrast certain pairs of data.

Original: Contrastive learning contrasts positive pairs and negative pairs.

Original: Contrastive divergence contrasts data and MCMC samples.

Original: However, they differ in the aspect that contrastive learning is inherently a discriminative learning method (in contrastive learning, the model “predicts” the positive pair among negative pairs) and contrastive divergence is a generative learning method.

Original: In the context of unsupervised visual representation learning, there are several works which connect contrastive learning with generative learning [28, 29, 30].

Original: For instance, Chen et al.

Original: [30] combine contrastive learning with

Original: GANs to improve person re-identification.

Original: Zimmermann et al.

Original: [29] and Wang et al.

Original: [28] study the distributional properties of embeddings found by contrastive learning.

Original: Our work, for the first time, explores the synergy between contrastive learning (the discriminative term in (11)) and EBMs / contrastive divergence (the generative term in (11)) for learning visual representations.

Original: A.2 Relation to Contrastive Energy-Based Models (CEMs) [19]

Original: According to the terms introduced in the work of Wang et al.

Original: [19], our EBCLR model distribution (6) with τ = 1 is identical to the model distribution of Non-Parametric CEM ((4) in [19]).

Original: This is because the squared ℓ 2 norm between two unit-norm vectors z and z ′ is equivalent to the inner product between z and z ′ up to an additive constant.

Original: However, our work differs from the work of Wang et al.

Original: [19] in three aspects.

Original: Objective function.

Original: Wang et al.

Original: [19] optimizes the log-likelihood of the joint model distribution q θ ( v, v ′ )

Original: directly whereas we use Bayes’ rule to decompose the objective into two terms, the discriminative term and the generative term.

Original: This allows us to reweight the generative term to learn better visual representations (as explored in Section 4.3).

Original: Optimization method.

Original: Wang et al.

Original: [19] maximizes the log-likelihood of the joint model distribution directly via adversarial training.

Original: On the other hand, we use gradient ascent to optimize the discriminative term and contrastive divergence to optimize the generative term.

Original: Computation costs are similar, as adversarial training also requires inner iterations to generate adversarial examples at every model update step.

Original: Application of interest.

Original: Wang et al. [19] focuses on enhancing image generation, while we focus on learning visual representations useful for downstream tasks. Wang et al. [19] also establishes a connection between contrastive divergence and InfoNCE (Section 5.1 in [19]).
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: Specifically, Wang et al.

Original: [19] show that when we approximate the model marginal q θ ( v ) with the data distribution p ( v ) , MLE gradient for q θ ( v, v ′ ) becomes the InfoNCE loss gradient.

Original: Surprisingly, we find this holds in our framework as well.

Original: If we set q θ ( v ) = p ( v ) in (11), the generative term in (11) becomes independent of θ , so we end up optimizing only the discriminative term.

Original: We have shown that the discriminative term can be approximated by a contrastive learning objective, so we recover the result of Wang et al.

Original: In this sense, the derivations in Section 3 of our paper generalizes the connection discovered in the work of Wang et al.

Original: [19], as we work with weaker assumptions (we do not set q θ ( v ) =

Original: p ( v ) in (11)).

Original: A.3 EBMs for Learning Useful Representations

Original: Ranzato et al.

Original: [14] proposes to train an EBM with an autoencoder structure by minimizing the sum of an encoding loss and a decoding loss to learn an energy surface.

Original: A byproduct is that the EBM learns a compressed representation of the data, which can be used for, e.g., denoising.

Original: Our work is similar in the aspect that we use a particular EBM architecture (a triplet network) and training the EBM with our proposed method results in a compressed representation of the data which is useful for downstream tasks, e.g., classification and transfer learning.

Original: In that sense, our work is also roughly related to Denoising Autoencoders (DAEs) [31] and Restricted

Original: Boltzmann Machines (RBMs) as well.

Original: Both are EBMs whose goal is to extract useful representations from data.

Original: However, in relating contrastive learning to EBMs, they about contrasting positive pairs pairs.

Original: A.4 Sample Efficiency of Generative Models

Original: We motivate the sample efficiency of EBCLR through the lens of discriminative and generative learning.

Original: We note that standard contrastive learning (e.g., SimCLR) optimizes a discriminative objective (the model must predict positive pairs) and EBCLR optimizes a generative objective (given a pair of images, the model must output how likely it is to be a positive pair).

Original: So, we can compare the performance of discriminative models and generative models.

Original: There is a long line of works which show (theoretically and / or empirically) that generative models can outperform discriminative ones.

Original: For instance, Efron [32] has theoretically shown that Normal Discriminant

Original: Analysis can be more efficient than logistic regression.

Original: Ng et al.

Original: [33] has also theoretically shown that in the small sample regime, naïve Bayes can be more resistant to overfitting than logistic regression (i.e., find better solutions when given a small number of training samples).

Original: There are also works [8, 9, 10] that consider optimizing a weighted combination of a discriminative loss and a generative loss, just like our EBCLR objective (11).

Original: They find that with an appropriate weight, the model can outperform discriminative models.

Original: In particular, Larochelle et al.

Original: [9] finds this hybrid approach to be beneficial in the limited training data setting.

Original: Grathwohl et al.

Original: [10] uses this hybrid approach to train a classifier and find that the classifier exhibits excellent calibration, unlike other calibration methods that require additional training data.

Original: We can also address the efficiency of generative models from the perspective of generative pre-training.

Original: Bengio et al.

Original: [34] finds that pre-training a network with a generative loss brings the network weights towards a good local minimum, thus bringing better generalization.

Original: Erhan et al.

Original: [35] finds that generative pre-training acts as a regularizer and that this regularization effect can be beneficial when we have a small training set.

Original: Another explanation specific to EBCLR is that the generative loss has an effect orthogonal to the effect of the discriminative loss.

Original: To minimize the generative loss, the model needs to place low energy on data and high energy elsewhere.

Original: This can be achieved only when the model has features that accurately capture the manifold of the data.

Original: When we have a small training set, the model may easily solve the discriminative objective and overfit.

Original: By providing an auxiliary generative task, the overfitting can be mitigated.

Original: An analogous effect with Joint Energy-Based Models is demonstrated by Zhao et al.

Original: B Psuedocodes

Original: Algorithm

Original: 1: Input: Energy E θ , initialization point ˜ v , count κ ˜ v , step size α , number of iterations T , proximity parameter δ , decay parameter K , variance bounds σ 2min , σ 2max 2: σ = σ min + ( σ max − σ min ) · [1 − κ ˜ v /K ]

Original: + 3: for t = 0 , 1 , 2 , . . .

Original: T − 1 do 4:

Original: Sample ϵ ∼ N (0 , σ 2 ) 5: ˜ v ← ˜ v − α · clamp {∇ v E θ

Original: (˜ v ) , δ } + ϵ 6: end for 7: κ ˜ v ← κ ˜ v +

Original: Calculate

Original: Sample

Original: Calculate E v

Original: { v

Original: Update { v

Original: MSGLD

Original: Calculate

Original: Update θ gradient ascent 11:

Original: Update with MSGLD

Original: 12: end while

Original: C Missing Proofs C.1 Proof of Theorem

Original: Proof.

Original: We first observe that

Original: [Equation]

Original: which proves the relation (12).

Original: We also have log q θ (

Original: [Equation]

Original: and so

Original: [Equation]

Original: Taking expectation w.r.t.

Original: p ( v ) establishes the relation (13).

Original: C.2 Another Theoretical Justification for EBCLR

Original: Here we provide another theoretical justification on how solving the EBCLR objective (11) with λ = 1 causes q θ ( v, v ′ ) to approximate p ( v, v ′ ) .

Original: It is known that optimizing the first term of (11) with (18) in place of q θ ( v ′ | v ) will cause (18) to approximate p ( v ′ | v ) [37].

Original: Also, optimizing the second term of (11) with (15) or (17) in place of q θ ( v ) will cause (15) or (17) to be proportional to p ( v ) (indeed, in Table 10, we see SGLD samples of the EBCLR marginal q θ ( v ) approximated by (17) achieve a non-trivial FID score).

Original: So, the product of (15) or (17) with (18) will be proportional to p ( v, v ′ ) .

Original: Moreover, by construction, the product of (15) or (17) with (18) is (approximately) q θ ( v, v ′ ) .

Original: Hence, optimizing the EBCLR objective (11) will cause q θ ( v, v ′ ) to model p ( v, v ′ ) .

Original: D Complete Training and Evaluation Details

Original: Baseline methods and datasets.

Original: The baseline methods are SimCLR, MoCo v2, SimSiam, and BYOL.

Original: The hyper-parameters are chosen closely following the original works [4, 12, 7, 6].

Original: We use four datasets: MNIST [23], Fashion MNIST (FMNIST)

Original: [24], CIFAR10, and CIFAR100 [25].

Original: Data transformation.

Original: For fair comparison, we use the stochastic data transformation proposed by Chen et.

Original: al [4] for all methods.

Original: In the case of grayscale images, we only use the random cropping augmentation.

Original: For EBCLR, we also add small Gaussian noise N (0 , 0 .

Original: 03 2 ) to stabilize training [16, 10, 15].

Original: We remark that Gaussian noise of standard deviation 0 .

Original: 03 is nearly invisible to the human eye.

Original: DNN architecture.

Original: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is the projection network.

Original: Rather than directly using the output of f θ for downstream tasks, we follow previous works [4, 5, 1, 2, 3, 6, 7] and use the output of ϕ θ instead.

Original: In our experiments, we set ϕ θ to be a ResNet-18 [26] up to the global average pooling layer.

Original: The architecture of π θ and π θ to be a 2-layer MLP with output convergence of SGLD.

Original: For the baseline methods, we use the settings proposed in the original works while keeping the backbone fixed to be ResNet-18.

Original: SGLD settings.

Original: The default parameters for MSGLD are |B| =

Original: 50 k , ρ = 0 .

Original: 2 , T = 10 , α = 0 .

Original: 05 , δ = 1 . 0 , K = 3 , σ min = 0 .

Original: 01 , and σ max = 0 .

Original: We use ρ = 0 .

Original: 4 for MNIST, ρ = 0 .

Original: 6 for FMNIST, and ρ = 0 . 2 for CIFAR10 and CIFAR100.

Original: At the beginning of training, we fill the buffer B entirely with proposal distribution samples.

Original: Training.

Original: The temperature is τ = 0 .

Original: 1 for EBCLR.

Original: For EBCLR, as we cannot use batch normalization, it is difficult to apply SGD with momentum.

Original: Hence, we use the Adam optimizer [38].

Original: The learning rate is 0 .

Original: if the batch size is 128 and 0 .

Original: if the batch size is smaller than 128 .

Original: We also weakly regularize the squared ℓ 2 -norm of the outputs of f θ with weight 0 .

Original: For the baseline methods, we closely follow the settings of previous works [4, 12, 27].

Original: [-Specifically, we use SGD with momentum 0.9 and learning rate of 0 . 
-]
Revised: [-Specifically, we use SGD with momentum 0.9 and learning rate of 0 . 
-]

Original: · ( batch size / 256) and weight decay 0 . 0001 .

Original: Each model is trained with a single GeForce RTX 3090.

Original: Evaluation.

Original: We assess the quality of the representations by training a linear classifier on top of frozen ϕ θ .

Original: The linear classifier is trained with Adam for 200 epochs with batch size 512 .

Original: The learning rate η is found by grid search for log 10 η in [ − 4 , − 1] .

Original: E Additional Experiments E.1 Section 4.1 with Longer Training

Original: We repeat the experiments in Section 4.1 by training the models for 200 epochs.

Original: For EBCLR, the number of SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.

Original: Tables 4 and 5 describe the results.

Original: We again note that EBCLR consistently beats all baseline methods and demonstrates high efficiency.

Original: [Table]

Original: [Table]

Original: E.2 Section 4.2 with Longer Training

Original: We repeat the experiments in Section 4.2 by training the models for 200 epochs.

Original: For EBCLR, the number of SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.

Original: Table 6 describes the results.

Original: We observe similar patterns as in the case of training for 100 epochs.

Original: In the case of CIFAR10, SimCLR seems to have a pattern agnostic to the batch size.

Original: However, we note that there is a significant gap between the performance of SimCLR with batch size 256 (see Table 4) and batch sizes in { 16 , 64 , 128 } .

Original: Interestingly, for SimCLR with small batch sizes, we observe that training longer with small batch sizes leads to worse performance than training only for 100 epochs.

Original: This is particularly prominent on CIFAR10 and CIFAR (compare Tables 3 and 6).

Original: We speculate that this is because, with small batches, SimCLR is unable to provide hard negatives, and this leads to overfitting of the DNN.

Original: On the other hand, EBCLR does not suffer from this phenomenon.

Original: This again demonstrates the importance of the generative term.

Original: [Table] Efficiency of SimCLR relative to EBCLR is calculated by the following formula: (number of epochs used by EBCLR with the same batch size to reach the final accuracy of SimCLR) / (total number of training epochs). E.3 Comparison of Approximations
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: We can either use Equation (15) or Equation (17) when approximating the generative term.

Original: Figure 5 shows results for both approximations on CIFAR10.

Original: We observe identical performance for both approximations, which implies that the generative term is robust to the number of samples used in the calculation of the empirical mean.

Original: This justifies our choice of using the simpler Equation (17) to approximate the marginal distribution q ( v ) .

Original: E.4 Evaluation with KNN Classifiers

Original: We also reproduce Tables 1,

Original: [Table]

Original: and 3 using K-nearest neighbour (KNN) classifiers.

Original: We followed the evaluation protocol of Caron et al.

Original: [39] and used a weighted KNN classifier with K = 20 .

Original: The results are given below.

Original: In

Original: Table 7, except for the case of BYOL on MNIST, we see that EBCLR beats all baselines again by a non-trivial margin.

Original: EBCLR is also shown to be sample efficient.

Original: In Table 8, we see EBCLR outperforms all baselines on the task of transfer learning.

Original: In Table 9, we see EBCLR is robust to small batch sizes.

Original: This leads to the lower efficiency of SimCLR compared to EBCLR.

Original: On CIFAR10 and CIFAR100, relative efficiency does not decrease below 0.05 since we saved checkpoints every 5 epochs.

Original: We can reasonably expect SimCLR to have lower relative efficiency at smaller batch sizes if we had saved checkpoints every epoch.

Original: [Table]

Original: [Table]

Original: [Table]

Original: E.5 Generative Performance of EBCLR

Original: In this section, we explore the effect of the discriminative loss on the generative performance of EBCLR.

Original: To this end, we solved

Original: [Equation]

Original: (23) for various values of γ on FMNIST.

Original: Recall that in the original EBCLR objective, weight λ comes in front of the generative term.

Original: We then measured the Fréchet Inception Distance (FID, lower is better) between samples from the true marginal p ( v ) and SGLD samples from the EBM marginal q θ ( v ) approximated by (19) / (17).

Original: The results are shown in Table 10.

Original: Interestingly, we found using an appropriate γ > 0 led to improvements in FID over γ = 0 .

Original: In other words, contrastive learning did help improve the generative performance of the model.

Original: However, using an excessively large γ led to a deterioration of the performance.

Original: This is analogous to the result of Section 4.3 where we observed using an appropriate weight on the generative term led to improvements over using only the discriminative term (contrastive loss).

Original: The difference is that to achieve optimal generative performance, the generative term weight needs to be larger (i.e., γ < 1 ) whereas to achieve optimal discriminative performance, the discriminative term weight needs to be larger (i.e., λ < 1 in (11)).

Original: We speculate that using a larger model could mitigate this trade-off, but this topic is beyond the scope of this work.

Original: [Table]
Revised: [N/A]

Original: F A Discussion on Making EBCLR Scalable

Original: We could consider three alternative losses / methods for training EBCLR: noise contrastive estimation (NCE)

Original: [40], VERA [41], and score matching [42].

Original: Although the NCE loss does not depend on MCMC, it requires specification of a noise distribution whose normalized density can be easily evaluated.

Original: In most cases, such noise distribution samples are easily distinguishable from natural image samples and lead to the density chasm problem [43], making optimization difficult.

Original: Telescoping density-ratio estimation [43] attempts to mitigate this problem, but it does not seem to scale well to large data.

Original: Flow contrastive estimation [44] uses a flow model to define the noise distribution and trains a flow model along with the EBM.

Original: Though flow contrastive estimation requires an auxiliary flow model, we believe this is the most promising alternative to MCMC.

Original: In a similar sense, VERA, which uses a generator to sample from the EBM, seems promising as well.

Original: Yet, flow contrastive estimation and VERA also did not demonstrate scalability to large-scale datasets such as ImageNet.

Original: Score matching matches the gradient of model log-density to the gradient of data log-density, so it does not rely on MCMC.

Original: However, it requires calculation of the Hessian, so in the deep learning setting, this approach is also quite expensive.