Original: Self-Supervised Learning via Maximum Entropy Codings
Revised: Self-Supervised Learning via Maximum Entropy Coding

Original: Abstract
Revised: Abstract

Original: A mainstream type of current self-supervised learning methods pursues a generalpurpose representation that can be well transferred to downstream tasks, typicallyby optimizing on a given pretext task such as instance discrimination.
Revised: A mainstream type of current self-supervised learning methods pursues a generalpurpose representation that can be well transferred to downstream tasks, typicallyby optimizing on a given pretext task such as instance discrimination.

Original: In thiswork, we argue that existing pretext tasks inevitably introduce biases into thelearned representation, which in turn leads to biased transfer performance on various downstream tasks.
Revised: In thiswork, we argue that existing pretext tasks inevitably introduce biases into thelearned representation, which in turn leads to biased transfer performance on various downstream tasks.

Original: To cope with this issue, we propose Maximum Entropy
Revised: To cope with this issue, we propose Maximum Entropy

Original: Coding ( MEC ), a more principled objective that explicitly optimizes on the structure of the representation, so that the learned representation is less biased andthus generalizes better to unseen downstream tasks.
Revised: Coding ( MEC ), a more principled objective that explicitly optimizes on the structure of the representation, so that the learned representation is less biased andthus generalizes better to unseen downstream tasks.

Original: Inspired by the principleof maximum entropy in information theory, we hypothesize that a generalizablerepresentation should be the one that admits the maximum entropy among allplausible representations.
Revised: Inspired by the principleof maximum entropy in information theory, we hypothesize that a generalizablerepresentation should be the one that admits the maximum entropy among allplausible representations.

Original: To make the objective end-to-end trainable, we proposeto leverage the minimal coding length in lossy data coding as a computationallytractable surrogate for the entropy, and further derive a scalable reformulation ofthe objective that allows fast computation.
Revised: To make the objective end-to-end trainable, we proposeto leverage the minimal coding length in lossy data coding as a computationallytractable surrogate for the entropy, and further derive a scalable reformulation ofthe objective that allows fast computation.

Original: Extensive experiments demonstratethat MEC learns a more generalizable representation than previous methods basedon specific pretext tasks.
Revised: Extensive experiments demonstratethat MEC learns a more generalizable representation than previous methods basedon specific pretext tasks.

Original: It achieves state-of-the-art performance consistently onvarious downstream tasks, including not only ImageNet linear probe, but alsosemi-supervised classification, object detection, instance segmentation, and objecttracking.
Revised: It achieves state-of-the-art performance consistently onvarious downstream tasks, including not only ImageNet linear probe, but alsosemi-supervised classification, object detection, instance segmentation, and objecttracking.

Original: Interestingly, we show that existing batch-wise ( e.g. , SimSiam [14])and feature-wise ( e.g. , Barlow Twins [75]) objectives could be seen equivalent tolow-order approximations of MEC.
Revised: Interestingly, we show that existing batch-wise ( e.g. , SimSiam [14])and feature-wise ( e.g. , Barlow Twins [81]) objectives could be seen equivalent tolow-order approximations of MEC.

Original: Code will be released.
Revised: Code will be released.

Original:  Introduction
Revised:  Introduction

Original: Self-supervised learning (SSL) aims to learn rich and meaningful representations without relyingon human annotations.
Revised: Self-supervised learning (SSL) aims to learn rich and meaningful representations without relyingon human annotations.

Original: Pursuing general-purpose representations, SSL models are typically used aspre-trained weights for providing a good initialization to downstream tasks.
Revised: Pursuing general-purpose representations, SSL models are typically used aspre-trained weights for providing a good initialization to downstream tasks.

Original: In this sense, SSL [12,27, 10, 25, 15, 14, 75, 26] has seen great progress in computer vision and can achieve competitive oreven better performance on various downstream tasks compared to its supervised counterparts.
Revised: In this sense, SSL [12,29, 10, 27, 15, 14, 81, 28] has seen great progress in computer vision and can achieve competitive oreven better performance on various downstream tasks compared to its supervised counterparts.

Original: At the core of current SSL methods is the design of pretext tasks .
Revised: At the core of current SSL methods is the design of pretext tasks .

Original: A pretext task is a (usuallyhand-crafted) learning objective in which the supervision signals could be mined from the data itself, with the aim of applying the learned representation to other downstream tasks.
Revised: A pretext task is a (usuallyhand-crafted) learning objective in which the supervision signals could be mined from the data itself, with the aim of applying the learned representation to other downstream tasks.

Original: Early attempts of pretext tasks typically discard a certain property of the input data and then force the model to predictthe discarded property.
Revised: Early attempts of pretext tasks typically discard a certain property of the input data and then force the model to predictthe discarded property.

Original: For instance, one can convert an RGB image to a gray-scale one, and train themodel to predict the original color [76, 77]; or apply random rotation to image patches and ask themodel to predict the rotation angles [24].
Revised: For instance, one can convert an RGB image to a gray-scale one, and train themodel to predict the original color [82, 83]; or apply random rotation to image patches and ask themodel to predict the rotation angles [25].

Original: In both cases, to fulfill the pretext task, the model must learn a meaningful representation related to object texture, shape or even category, therefore the learned representation transfers well to downstream tasks related to these features
Revised: In both cases, to fulfill the pretext task, the model must learn a meaningful representation related to object texture, shape or even category, therefore the learned representation transfers well to downstream tasks related to these features

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: SimCLR
Revised: SimCLR

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: MoCo-v
Revised: MoCo-v

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: BYOL
Revised: BYOL

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: SwAV
Revised: SwAV

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: SimSiam
Revised: SimSiam

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: BarlowTwins
Revised: BarlowTwins

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: MEC
Revised: MEC

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: InsDis
Revised: InsDis

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: SimCLR
Revised: SimCLR

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: MoCo-v
Revised: MoCo-v

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: InfoMin
Revised: InfoMin

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: BYOL
Revised: BYOL

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: SwAV
Revised: SwAV

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: MEC
Revised: MEC

Original: Despite successes of existing pretext tasks, we find that they inevitably introduce biases into thelearned representation, which conflict with the original aim of “general-purpose”.
Revised: Despite successes of existing pretext tasks, we find that they inevitably introduce biases into thelearned representation, which conflict with the original aim of “general-purpose”.

Original: For example,representations trained with the most prevalent image-level task, instance discrimination [19, 70], arefound biased to image-level tasks such as image classification, while by contrast degenerate in patchor pixel-level tasks like object detection and semantic segmentation [71].
Revised: For example,representations trained with the most prevalent image-level task, instance discrimination [19, 75], arefound biased to image-level tasks such as image classification, while by contrast degenerate in patchor pixel-level tasks like object detection and semantic segmentation [76].

Original: Even being transferredto image-level tasks, such representations still suffer from domain gaps in cases of different datadistributions [23], e.g. , classification on unseen categories other than ImageNet objects.
Revised: Even being transferredto image-level tasks, such representations still suffer from domain gaps in cases of different datadistributions [24], e.g. , classification on unseen categories other than ImageNet objects.

Original: In this work, we are curious of what makes for generalizable representations , and pursue an explicitoptimization with a criterion that directly measures the structure of representations, minimizingthe biases brought by the pretext task.
Revised: In this work, we are curious of what makes for generalizable representations , and pursue an explicitoptimization with a criterion that directly measures the structure of representations, minimizingthe biases brought by the pretext task.

Original: To this end, we propose Maximum Entropy Coding (MEC).
Revised: To this end, we propose Maximum Entropy Coding (MEC).

Original: Inspired by the principle of maximum entropy in information theory, the basic hypothesis in MECis that a generalizable representation should be the one that admit the maximum entropy among allplausible representations.
Revised: Inspired by the principle of maximum entropy in information theory, the basic hypothesis in MECis that a generalizable representation should be the one that admit the maximum entropy among allplausible representations.

Original: Accordingly, optimizing towards maximum entropy leads to representationswith good generalization capacity.
Revised: Accordingly, optimizing towards maximum entropy leads to representationswith good generalization capacity.

Original: The main challenge confronting us is that it is difficult, andcomputationally expensive, if possible, to estimate the distribution of a given representation (usually afinite set of high-dimensional vectors), so in turn it is difficult to estimate the entropy.
Revised: The main challenge confronting us is that it is difficult, andcomputationally expensive, if possible, to estimate the distribution of a given representation (usually afinite set of high-dimensional vectors), so in turn it is difficult to estimate the entropy.

Original: To cope with thisissue, we replace the optimization objective from the originally defined entropy to a computationallytractable surrogate, i.e. , the necessary number of bits needed to encode the representations vialossy data coding [16].
Revised: To cope with thisissue, we replace the optimization objective from the originally defined entropy to a computationallytractable surrogate, i.e. , the necessary number of bits needed to encode the representations vialossy data coding [16].

Original: The log-determinant term costs the most computation in the coding lengthfunction.
Revised: The log-determinant term costs the most computation in the coding lengthfunction.

Original: By leveraging Taylor expansion of matrix, we further approximate this term with polynomialfunctions, leading to significant speed up and thus makes large-scale pre-training possible.
Revised: By leveraging Taylor expansion of matrix, we further approximate this term with polynomialfunctions, leading to significant speed up and thus makes large-scale pre-training possible.

Original: In contrast to previous SSL methods that mainly evaluate on ImageNet classification, we experimenton a wide variaty of vision tasks to show the good generalization capacity of MEC.
Revised: In contrast to previous SSL methods that mainly evaluate on ImageNet classification, we experimenton a wide variety of vision tasks to show the good generalization capacity of MEC.

Original: The consideredtasks span across not only image-level recognition on various data distributions, but also patch- orpixel-level tasks like object detection, instance segmentation, and object tracking.
Revised: The consideredtasks span across not only image-level recognition on various data distributions, but also patch- orpixel-level tasks like object detection, instance segmentation, and object tracking.

Original: We show MECgeneralizes well consistently across all tasks considered, while representations learned with previouspretext tasks usually perform well on closely related downstream tasks but degenerate on less relatedones (see Figure 1).
Revised: We show MECgeneralizes well consistently across all tasks considered, while representations learned with previouspretext tasks usually perform well on closely related downstream tasks but degenerate on less relatedones (see Figure 1).

Original: Besides empirical results, we find interesting equivalence between low-orderapproximations of MEC and
Revised: Besides empirical results, we find interesting equivalence between low-orderapproximations of MEC and

Original: existing batch-wise ( e.g. , SimSiam [14]) or feature-wise objectives ( e.g. ,
Revised: existing batch-wise ( e.g. , SimSiam [14]) or feature-wise objectives ( e.g. ,

Original: Barlow Twins [75]), which provides a new perspective for a unified understanding of prevalent SSLmethods.
Revised: Barlow Twins [81]), which provides a new perspective for a unified understanding of prevalent SSLmethods.

Original: In summary, the main contributions of our work are as follows:
Revised: In summary, the main contributions of our work are as follows:

Original: • To learn representations generalizable to various downstream tasks, we introduce the principle of
Revised: • To learn representations generalizable to various downstream tasks, we introduce the principle of

Original: maximum entropy into self-sueprvised learning.• We propose Maximum Entropy Coding ( MEC ), which explicitly optimizes on representationsbased on the principle of maximum entropy, and leverages the minimal coding length in lossydata coding as a computationally tractable surrogate for the entropy.
Revised: maximum entropy into self-supervised learning.• We propose Maximum Entropy Coding ( MEC ), which explicitly optimizes on representationsbased on the principle of maximum entropy, and leverages the minimal coding length in lossydata coding as a computationally tractable surrogate for the entropy.

Original: • We reformulate the log-determinant term in coding length function into a scalable form, which

Original: makes large-scale pre-training possible, and unifies existing batch-wise and feature-wise objectives as low-order approximations of our method.

Original: • We show MEC representation generalizes well on a wide variety of image- and video-baseddownstream tasks, achieving state-of-the-arts on most tasks considered.

Original: [Table]
Revised: [Table]

Original: /AoLAwix8g9s/A1OmwFajnSlo3Pu1b33eBGjUlnWt1FYW9/Y3Cpul3Z29/YPzMOjjgxjgUkbhywUPQ9JwignbUUVI71IEBR4jHS9yU3mdx+IkDTkLTWLiBOgEac+xUhpyTVPK4MAqTFGLLlL3aSBBAunsDWlXKYV1yxbVWsOuErsnJRBjqZrfg2GIY4DwhVmSMq+bUXKSZBQFDOSlgaxJBHCEzQifU05Coh0kvkXKTzXyhD6odDFFZyrvycSFEg5CzzdmZ0sl71M/M/rx8q/dhLKo1gRjheL/JhBFcIsEjikgmDFZpogLKi+FeIxEggrHVxJh2Avv7xKOrWqfVm172vleiOPowhOwBm4ADa4AnVwC5qgDTB4BM/gFbwZT8aL8W58LFoLRj5zDP7A+PwBXTKYfA==</latexit> L BarlowTwins 
Revised: /AoLAwix8g9s/A1OmwFajnSlo3Pu1b33eBGjUlnWt1FYW9/Y3Cpul3Z29/YPzMOjjgxjgUkbhywUPQ9JwignbUUVI71IEBR4jHS9yU3mdx+IkDTkLTWLiBOgEac+xUhpyTVPK4MAqTFGLLlL3aSBBAunsDWlXKYV1yxbVWsOuErsnJRBjqZrfg2GIY4DwhVmSMq+bUXKSZBQFDOSlgaxJBHCEzQifU05Coh0kvkXKTzXyhD6odDFFZyrvycSFEg5CzzdmZ0sl71M/M/rx8q/dhLKo1gRjheL/JhBFcIsEjikgmDFZpogLKi+FeIxEggrHVxJh2Avv7xKOrWqfVm172vleiOPowhOwBm4ADa4AnVwC5qgDTB4BM/gFbwZT8aL8W58LFoLRj5zDP7A+PwBXTKYfA==</latexit> L BarlowTwins asym.

Revised: • We reformulate the log-determinant term in coding length function into a scalable form, whichmakes large-scale pre-training possible, and unifies existing batch-wise and feature-wise objec-

Original: [Figure]
Revised: tives as low of our • We show MEC representation generalizes well on a wide variety of image- and video-based downstream tasks, achieving state-of-the-arts on most tasks considered.

Original:  Method
Revised:  Method

Original: [Figure] In this section, we start with illustrating the maximum entropy principle, and then introduce acomputationally tractable surrogate of the information entropy for high-dimensional vectors.
Revised: In this section, we start with illustrating the maximum entropy principle, and then introduce acomputationally tractable surrogate of the information entropy for high-dimensional vectors.

Original: We thenpresent a scalable reformulation of the proposed surrogate, which makes large-scale training possible.
Revised: We thenpresent a scalable reformulation of the proposed surrogate, which makes large-scale training possible.

Original: And we further incorporate the view consistency prior for maximum entropy coding.
Revised: And we further incorporate the view consistency prior for maximum entropy coding.

Original: Finally, wedemonstrate how the proposed method can unify existing batch-wise and feature-wise SSL objectives.
Revised: Finally, wedemonstrate how the proposed method can unify existing batch-wise and feature-wise SSL objectives.

Revised: Please refer to Appendix H for more details about the proofs in this section.

Original: 2.1 Maximum Entropy Coding
Revised: 2.1 Maximum Entropy Coding

Original: [Figure]

Original: The maximum entropy principle.
Revised: The maximum entropy principle.

Original: The main purpose of this work is to improve the generalizationcapacity of self-supervised learning representations across unseen downstream tasks and data distributions, and reduce the biases brought by specifically designed pretext tasks as much as possible.
Revised: The main purpose of this work is to improve the generalizationcapacity of self-supervised learning representations across unseen downstream tasks and data distributions, and reduce the biases brought by specifically designed pretext tasks as much as possible.

Original: This naturally raises a question, i.e. , what makes for a generalizable representation?
Revised: This naturally raises a question, i.e. , what makes for a generalizable representation?

Original: To answer thisquestion, we are particularly inspired by the maximum entropy principle in information theory, whichstates that the probability distribution that best represents the current state of knowledge about asystem is the one with largest entropy, given a testable information (such as accuracy) and in this wayno additional bias or assumptions is introduced [34, 35, 51].
Revised: To answer thisquestion, we are particularly inspired by the maximum entropy principle in information theory, whichstates that the probability distribution that best represents the current state of knowledge about asystem is the one with largest entropy, given a testable information (such as accuracy) and in this wayno additional bias or assumptions is introduced [36, 37, 55].

Original: We therefore hypothesis that a generalizable representation is the one that has the maximum entropy among all plausible representations.
Revised: We therefore hypothesis that a generalizable representation is the one that has the maximum entropy among all plausible representations.

Original: Intuitively, if we are able to express the entropy in a closed form, the maximum entropy principlethen can serve as an optimization objective and supervise the representation learning.
Revised: Intuitively, if we are able to express the entropy in a closed form, the maximum entropy principlethen can serve as an optimization objective and supervise the representation learning.

Original: Minimal coding length as a surrogate for entropy.
Revised: Minimal coding length as a surrogate for entropy.

Original: Entropy is originally defined on probabilitydistributions [54], i.e. , H ( z ) ≜ − (cid:82) p ( z ) log p ( z ) dz , for continuous random variables.
Revised: Entropy is originally defined on probabilitydistributions [58], i.e. , H ( z ) ≜ − (cid:82) p ( z ) log p ( z ) dz , for continuous random variables.

Original: However,it is very difficult to estimate the true distributions p ( z ) of a representation [6, 46], from a finiteset of high dimensional vectors Z = (cid:2)
Revised: However,it is very difficult to estimate the true distributions p ( z ) of a representation [6, 50], from a finiteset of high dimensional vectors Z = (cid:2)

Original: z 1 , z 2 , . .
Revised: z 1 , z 2 , . .

Original: , z m (cid:3) ∈ R d × m .
Revised: , z m (cid:3) ∈ R d × m .

Original: A handy fact is that entropy isconceptually equivalent to the minimal number of bits required to encode the data losslessly, so theminimal lossless coding length could be used to represent the entropy.
Revised: A handy fact is that entropy isconceptually equivalent to the minimal number of bits required to encode the data losslessly, so theminimal lossless coding length could be used to represent the entropy.

Original: However, lossless coding ofcontinuous random variables is infeasible in our case since it often requires an infinite number of bits,breaking the numerical stability.
Revised: However, lossless coding ofcontinuous random variables is infeasible in our case since it often requires an infinite number of bits,breaking the numerical stability.

Original: Instead, we exploit the coding length in lossy data coding [16] as acomputationally tractable surrogate for the entropy of continuous random variables.
Revised: Instead, we exploit the coding length in lossy data coding [16] as acomputationally tractable surrogate for the entropy of continuous random variables.

Original: Given a set ofsamples Z , the minimal number of bits needed to encode Z subject to a distortion ϵ is given by thefollowing coding length function [39, 62]:
Revised: Given a set ofsamples Z , the minimal number of bits needed to encode Z subject to a distortion ϵ is given by thefollowing coding length function [43, 67]:

Original: [Equation]
Revised: [Equation]

Original: where I m denotes the identity matrix with dimension m , and ϵ is the upper bound of the expecteddecoding error between z ∈ Z and the decoded (cid:98) z , i.e. , E [ ∥ z − (cid:98)
Revised: where I m denotes the identity matrix with dimension m , and ϵ is the upper bound of the expected

Original: z ∥ 2 ] ≤ ϵ .
Revised: [Figure] [Equation]

Original: We note that the computation of log-determinant of high dimensional matrix in Equation (1) is highlyexpensive and may cause numerically unstable results for ill-conditioned matrix, which inhibits its application to large-scale pre-training ( e.g. , over 1 million images).
Revised: We note that the computation of log-determinant of high dimensional matrix in Equation (1) is highlyexpensive and may cause numerically unstable results for ill-conditioned matrix, which inhibits its application to large-scale pre-training ( e.g. , over 1 million images).

Original: Therefore, a scalable and stable reformulation of Equation (1) is required.
Revised: Therefore, a scalable and stable reformulation of Equation (1) is required.

Original: We first rewrite Equation (1) as L = µ log det (cid:16) I m + λ Z ⊤ Z (cid:17) ,where µ = m + dand λ = dmϵ 2 .
Revised: We first rewrite Equation (1) as L = µ log det (cid:16) I m + λ Z ⊤ Z (cid:17) ,where µ = m + dand λ = dmϵ 2 .

Original: Utilizing the identical equation det(exp( A ))
Revised: Utilizing the identical equation det(exp( A ))

Original: =
Revised: =

Original: exp(Tr( A ))
Revised: exp(Tr( A ))

Original: [31],we obtain L =
Revised: [33],we obtain L =

Original: Tr (cid:16) µ log (cid:16) I m + λ Z ⊤ Z (cid:17)(cid:17) , where Tr stands for the trace of the matrix.
Revised: Tr (cid:16) µ log (cid:16) I m + λ Z ⊤ Z (cid:17)(cid:17) , where Tr stands for the trace of the matrix.

Original: Finally, we [Figure] apply Taylor series expansion to expand the logarithm of the matrix and obtain
Revised: Finally, weapply Taylor series expansion to expand the logarithm of the matrix and obtain

Original: L =
Revised: [Equation]

Original: Tr (cid:32) µ ∞ (cid:88) k
Revised: [Figure] [Equation] [Figure]

Original: = ( − 1) k
Revised: [Equation]

Original: + k (cid:16) λ Z ⊤ Z (cid:17) k (cid:33) , (2)   2048 dim 0. t i m e ( m s ) ( l o g - s c a l e ) [0.24%] [0.18%] [0.11%] [0.07%] approx.
Revised: [Figure]

Original: origin with convergence condition: (cid:13)(cid:13)(cid:13) λ
Revised: with convergence condition: (cid:13)(cid:13)(cid:13) λ

Original: Z ⊤ Z (cid:13)(cid:13)(cid:13) 2 < 1 , and this canbe achieved by adjusting the hyperparameter ϵ (detailedin Section 2.2).
Revised: Z ⊤ Z [Equation] 2 < 1 , and this canbe achieved by adjusting the hyperparameter ϵ (detailedin Section 2.2).

Original: Compared to Equation (1), there are onlymatrix multiplication and addition in Equation (2), whichsignificantly speeds up the computation process and avoidsthe numerical unstability.
Revised: Compared to Equation (1), there are onlymatrix multiplication and addition in Equation (2), whichsignificantly speeds up the computation process and avoidsthe numerical unstability.

Original: To verify this, in Figure 3, wemake a comparison of running time and relative approximation error between Equation (1) and (2) (using the firstfour terms).
Revised: To verify this, in Figure 3, wemake a comparison of running time and relative approximation error between Equation (1) and (2) (using the firstfour terms).

Original: The results show that our reformulation canapproximate the original coding length function with negligible error (all errors are well below 0 .
Revised: The results show that our reformulation canapproximate the original coding length function with negligible error (all errors are well below 0 .

Original: 5% ), and acceleratethe computation considerably (over 50 × acceleration forall cases), thus making large-scale pre-training possible.
Revised: 5% ), and acceleratethe computation considerably (over 50 × acceleration forall cases), thus making large-scale pre-training possible.

Original: 2.2 Combining with the View Consistency Prior
Revised: 2.2 Combining with the View Consistency Prior

Original: It should be noted that the necessary premise of the maximum entropy principle is that testableinformation is given as prior.
Revised: It should be noted that the necessary premise of the maximum entropy principle is that testableinformation is given as prior.

Original: For example, the testable information could be the accuracy of apredictive model: the most generalizable model should be the one with maximum entropy, but itis only when a group of models reaches a given accuracy.
Revised: For example, the testable information could be the accuracy of apredictive model: the most generalizable model should be the one with maximum entropy, but itis only when a group of models reaches a given accuracy.

Original: Otherwise, simply optimizing towardsmaximum entropy will lead to trivial solutions such as uniform distributions.
Revised: Otherwise, simply optimizing towardsmaximum entropy will lead to trivial solutions such as uniform distributions.

Original: In Equation (1), the prioris not considered.
Revised: In Equation (1), the prioris not considered.

Original: To introduce a prior, we employ a common practice in SSL by augmenting Z intotwo different views Z 1 and Z 2 .
Revised: To introduce a prior, we employ a common practice in SSL by augmenting Z intotwo different views Z 1 and Z 2 .

Original: More specifically, given a set of images D , an image x is sampleduniformly from D , and two augmented views x 1 and x 2 are obtained from x via a distribution ofdata augmentations T .
Revised: More specifically, given a set of images D , an image x is sampleduniformly from D , and two augmented views x 1 and x 2 are obtained from x via a distribution ofdata augmentations T .

Original: Then they are fed to an encoder f , consisting of a backbone and a projectornetwork, which produces ℓ 2 -normalized embeddings of z 1 and z 2 .
Revised: Then they are fed to an encoder f , consisting of a backbone and a projectornetwork, which produces ℓ 2 -normalized embeddings of z 1 and z 2 .

Original: For a batch of m images, we have
Revised: For a batch of m images, we have

Original: Z 1 = (cid:2) z 11 , z 21 , . . .
Revised: Z 1 = (cid:2) z 11 , z 21 , . . .

Original: , z m 1 (cid:3) and similarly for Z 2 , which are two observations of the same Z .
Revised: , z m 1 (cid:3) and similarly for Z 2 , which are two observations of the same Z .

Original: MEC aimsto minimize the following loss:
Revised: MEC aimsto minimize the following loss:

Original: [Equation]
Revised: [Equation]

Original: where the same notations in Section 2.1 apply, and n is the order of Taylor expansion.
Revised: where the same notations in Section 2.1 apply, and n is the order of Taylor expansion.

Original: Comparedwith Equation (1), the formulation in (3) considers not only maximizing entropy, but also the viewconsistency prior mined from the data itself, therefore learning meaningful representations.
Revised: Comparedwith Equation (1), the formulation in (3) considers not only maximizing entropy, but also the viewconsistency prior mined from the data itself, therefore learning meaningful representations.

Original: As noted in Section 2.1, the convergence condition of Taylor expansion requires ∥ C ∥ 2 < 1 , where
Revised: As noted in Section 2.1, the convergence condition of Taylor expansion requires ∥ C ∥ 2 < 1 , where

Original: C = λ
Revised: C = λ

Original: Z ⊤ 1 Z 2 and λ = dmϵ 2 = 1 mϵ 2 d .
Revised: Z ⊤ 1 Z 2 and λ = dmϵ 2 = 1 mϵ 2 d .

Original: We show such condition can be strictly satisfied by setting ϵ 2 d =dm because of the inequality ∥ C ∥ 2 ≤ (cid:112) ∥ C ∥ 1 ∥ C ∥ ∞ < 1 .
Revised: We show such condition can be strictly satisfied by setting ϵ 2 d > 1because of the inequality ∥ C ∥ 2 ≤ (cid:112) ∥ C ∥ 1 ∥ C ∥ ∞ < 1 .

Original: In practice, we empirically find that the  Taylor expansion converges over a wide range of ϵ d (Figure 4(c)).
Revised: In practice, we empirically find that the 149 Taylor expansion converges over a wide range of ϵ d (Figure 4(c)).

Original: From the preliminary experimentson CIFAR-10 [37] (detailed in Appendix B), we also find that the distributions of representationsshow progressive finer granularity as ϵ d decreases (Figure 4(b)).
Revised: From the preliminary experimentson CIFAR-10 [39] (detailed in Appendix B), we also find that the distributions of representationsshow progressive finer granularity as ϵ d decreases (Figure 4(b)).

Original: This can be interpreted by thepractical meaning of the distortion ϵ d (Figure 4(a)), i.e. , a smaller ϵ d encourages the representationspace to be encoded in finer granularity (and hence more uniform).
Revised: This can be interpreted by thepractical meaning of the distortion ϵ d (Figure 4(a)), i.e. , a smaller ϵ d encourages the representationspace to be encoded in finer granularity (and hence more uniform).

Original: By contrast, a small ϵ d mightbreak the semantic structures of similar images ( i.e. , tolerance).
Revised: By contrast, a small ϵ d mightbreak the semantic structures of similar images ( i.e. , tolerance).

Original: Therefore, a good choice of ϵ d isneeded to compromise the uniformity and tolerance properties [65, 64] of representations, whichshares the same role as the temperature [70] term in constrastive learning.
Revised: Therefore, a good choice of ϵ d isneeded to compromise the uniformity and tolerance properties [70, 69] of representations, whichshares the same role as the temperature [75] term in contrastive learning.

Original: An overview of MEC is illustrated in Figure 2 and a PyTorch-like pseudocode is provided in
Revised: An overview of MEC is illustrated in Figure 2 and a PyTorch-like pseudocode is provided in

Original: Appendix A.
Revised: Appendix A.

Original: The algorithm describes the minimalist variant of MEC, which can be further improvedby integrating momentum encoder and asymmetric networks (detailed in experiments).
Revised: The algorithm describes the minimalist variant of MEC, which can be further improvedby integrating momentum encoder and asymmetric networks (detailed in experiments).

Original: 2.3 A Unified View of Batch-wise and Feature-wise SSL Objectives
Revised: 2.3 A Unified View of Batch-wise and Feature-wise SSL Objectives

Original: Current SSL methods based on Siamese networks can be roughly divided into two categories: batchwise methods [12, 27, 13, 15, 14, 10] and feature-wise methods [75, 5, 22, 32].
Revised: Current SSL methods based on Siamese networks can be roughly divided into two categories: batchwise methods [12, 29, 13, 15, 14, 10] and feature-wise methods [81, 5, 22, 34].

Original: The former aims tominimize the distance between augmented views of the same sample while maximizing the distancebetween different samples, which can be viewed as decorrelating the different features in a batch.
Revised: The former aims tominimize the distance between augmented views of the same sample while maximizing the distancebetween different samples, which can be viewed as decorrelating the different features in a batch.

Original: The latter, in contrast, tries to decorrelate the different vector components in the representation.
Revised: The latter, in contrast, tries to decorrelate the different vector components in the representation.

Original: Therelationship between them has not been fully understood.
Revised: Therelationship between them has not been fully understood.

Original: Our work builds bridges between these twotypes of methods through the following derivation:
Revised: Our work builds bridges between these twotypes of methods through the following derivation:

Original: [Equation]
Revised: [Equation]

Original: (cid:123)(cid:122) (cid:125) feature-wise , (4) which can be proved since Z ⊤ 1 Z 2 ∈ R m × m and Z 1 Z ⊤ 2 ∈ R
Revised: which can be proved since Z ⊤ 1 Z 2 ∈ R m × m and Z 1 Z ⊤ 2 ∈ R

Original: d × d have the same  eigenvalues.
Revised: d × d have the same nonzero eigenvalues.

Original: In Figure 2, under the framework of MEC, we show the equivalence between batch-wise and featurewise methods using two examples, SimSiam [14] and Barlow Twins [75].
Revised: In Figure 2, under the framework of MEC, we show the equivalence between batch-wise and featurewise methods using two examples, SimSiam [14] and Barlow Twins [81].

Original: By taking Taylor expansion(Equation (2)) of the left side of Equation (4) and before the trace operation, the diagonal elements ofthe leading term ( i.e. , µλ Z ⊤ 1 Z 2 ) measure the similarity between the views of the same images in abatch, and the objective of SimSiam [14] is equivalent to maximizing the trace of this term.
Revised: By taking Taylor expansion(Equation (2)) of the left side of Equation (4) and before the trace operation, the diagonal elements ofthe leading term ( i.e. , µλ Z ⊤ 1 Z 2 ) measure the similarity between the views of the same images in abatch, and the objective of SimSiam [14] is equivalent to maximizing the trace of this term.

Original: Similarly,the leading term of the right side expansion models the correlation between dimensions of the feature,and the objective of Barlow Twins [75] is equivalent to the second-order expansion of L MEC .
Revised: Similarly,the leading term of the right side expansion models the correlation between dimensions of the feature,and the objective of Barlow Twins [81] is equivalent to the second-order expansion of L MEC .

Original: Withthe above derivation, our method naturally subsumes the two different kinds of objectives as itslow-order expansions, and we show in experiments that better downstream task performance can beachieved with higher-order approximations.
Revised: Withthe above derivation, our method naturally subsumes the two different kinds of objectives as itslow-order expansions, and we show in experiments that better downstream task performance can beachieved with higher-order approximations.

Original:  Experiments
Revised:  Experiments

Original: We perform self-supervised pre-training using the proposed MEC on the training set of the ImageNet ILSVRC-2012 dataset [17].
Revised: We perform self-supervised pre-training using the proposed MEC on the training set of the ImageNet ILSVRC-2012 dataset [17].

Original: After pre-training, we conduct extensive experiments to examinethe learned representations on various tasks.
Revised: After pre-training, we conduct extensive experiments to examinethe learned representations on various tasks.

Original: The tasks considered include four image-level tasks:
Revised: The tasks considered include four image-level tasks:

Original: ImageNet linear probing, ImageNet semi-supervised classification, object detection, and instance segmentation; and five video-based tasks: single object tracking (SOT) [68], video object segmentation(VOS) [48], multi-object detection (MOT) [40], multi-object detection and segmentation (MOTS) 
Revised: ImageNet linear probing, ImageNet semi-supervised classification, object detection, and instance segmentation; and five video-based tasks: single object tracking (SOT) [73], video object segmentation(VOS) [52], multi-object detection (MOT) [44], multi-object detection and segmentation (MOTS) [68] [Table] [Table]

Original: [63]and pose tracking (PoseTrack) [1].
Revised: VOC detection tasks and Mask R-CNN [30] (1 × schedule) for COCO detection and instance segmentation tasks. All Faster/Mask R-CNN models are with the C4-backbone [74]. Bold entries are within 0.2 below the best. [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: MEC 77.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: and pose tracking (PoseTrack)

Original: An overview comparison with existing SSL representations acrossdifferent tasks is presented in Section 3.1.
Revised: An overview comparison with existing SSL representations acrossdifferent tasks is presented in Section 3.1.

Original: Moreover, in Section 3.4, we present extensive ablationsstudies to demonstrate the behaviors and properties of the proposed method.
Revised: Moreover, in Section 3.4, we present extensive ablationsstudies to demonstrate the behaviors and properties of the proposed method.

Original: All the implementationdetails of the aforementioned experiments are deferred to Appendix C.
Revised: All the implementationdetails of the aforementioned experiments are deferred to Appendix C.

Original: 3.1 Overview Comparison
Revised: 3.1 Overview Comparison

Original: We evaluate MEC together with multiple previous SSL representations on the four image-based tasksand the five video-based tasks, plot their performance ranking along a axis for a specific task and [Table] [Table] VOC detection tasks and Mask R-CNN [28] (1 × schedule) for COCO detection and instance segmentation tasks.
Revised: We evaluate MEC together with multiple previous SSL representations on the four image-based tasksand the five video-based tasks, plot their performance ranking along a axis for a specific task andobtain two sets of radar-like charts in Figure 1.

Original: All Faster/Mask R-CNN models are with the C4-backbone [69].

Original: Bold entries are within 0.2 below the best.

Original: [Table]

Original: obtain two sets of radar-like charts in Figure 1.

Original: Intuitively, it can be seen that the larger a polygon in aradar chart is, the better the corresponding representation generalize across downstream tasks.
Revised: Intuitively, it can be seen that the larger a polygon in aradar chart is, the better the corresponding representation generalize across downstream tasks.

Original: MECconsistently outperforms previous representations on both image- and video-based tasks considered,suggesting its good generalization capacity to various downstream tasks and data distributions.
Revised: MECconsistently outperforms previous representations on both image- and video-based tasks considered,suggesting its good generalization capacity to various downstream tasks and data distributions.

Original: Belowwe detail experiments on each task.
Revised: Belowwe detail experiments on each task.

Original: And additional experiment results can be found in Appendix D.
Revised: And additional experiment results can be found in Appendix D.

Original: 3.2 Evaluation on ImageNet
Revised: 3.2 Evaluation on ImageNet

Original: Linear probing.
Revised: Linear probing.

Original: We train a linear classifier on top of frozen representations of the pre-trained modelon the ImageNet training set, and report the top-1 accuracy on the ImageNet validation set, which is astandard and important protocol in SSL [77, 12, 27, 14, 75].
Revised: We train a linear classifier on top of frozen representations of the pre-trained modelon the ImageNet training set, and report the top-1 accuracy on the ImageNet validation set, which is astandard and important protocol in SSL [83, 12, 29, 14, 81].

Original: We evaluate the learned representationsof models with four different pre-training epochs following [14], and make comparisons in Table 1with state-of-the-art methods, whose results are reported in [14] and better than their original papers.
Revised: We evaluate the learned representationsof models with four different pre-training epochs following [14], and make comparisons in Table 1with state-of-the-art methods, whose results are reported in [14] and better than their original papers.

Original: Even so, our method MEC achieves highest accuracy under all pre-training epochs among thosemethods.
Revised: Even so, our method MEC achieves highest accuracy under all pre-training epochs among thosemethods.

Original: In particular, with only 200-epoch pre-training, MEC outperforms several 800-epochpre-trained methods, including SimCLR [12], SwAV [10] and SimSiam [14], which indicates the pretraining efficiency of our method.
Revised: In particular, with only 200-epoch pre-training, MEC outperforms several 800-epochpre-trained methods, including SimCLR [12], SwAV [10] and SimSiam [14], which indicates the pretraining efficiency of our method.

Original: Additionally, by incorporating other pre-training strategies, such asmulti-crop [10, 11], our method can attain even better results, which are detailed in Appendix D. {+[Table] MEC 56.
+}
Revised: Additionally, by incorporating other pre-training strategies, such asmulti-crop [10, 11], our method can attain even better results, which are detailed in Appendix D. {+[Table] MEC 56.
+}

Original: Semi-supervised classification.
Revised: Semi-supervised classification.

Original: We fine-tune the pre-trained model on a small subset of ImageNetfor classification task.
Revised: We fine-tune the pre-trained model on a small subset of ImageNetfor classification task.

Original: Specifically, we adopt the same fixed splits of 1% and 10% of ImageNettraining set as in [12] and report both top-1 and top-5 accuracies in Table 2.
Revised: Specifically, we adopt the same fixed splits of 1% and 10% of ImageNettraining set as in [12] and report both top-1 and top-5 accuracies in Table 2.

Original: The experiment resultsshow that our method consistently outperforms previous approaches in both 1% and 10% settings.
Revised: The experiment resultsshow that our method consistently outperforms previous approaches in both 1% and 10% settings.

Original: And we also note that all SSL methods surpass the supervised counterparts by a large margin ( e.g. ,over 20% accuracy using 1% labeled data), which emphasizes the importance of leveraging largeamount of unlabeled data.[Table] [Table]
Revised: And we also note that all SSL methods surpass the supervised counterparts by a large margin ( e.g. ,over 20% accuracy using 1% labeled data), which emphasizes the importance of leveraging largeamount of unlabeled data.

Original: 3.3 Transfer learning
Revised: 3.3 Transfer learning

Original: Object detection and instance segmentation.
Revised: Object detection and instance segmentation.

Original: We transfer the pre-trained model by end-to-endfine-tuning on object detection and instance segmentation tasks, following the common practice ofprevious methods [27, 13, 14, 25, 75].
Revised: We transfer the pre-trained model by end-to-endfine-tuning on object detection and instance segmentation tasks, following the common practice ofprevious methods [29, 13, 14, 27, 81].

Original: We report the results in Table 3.
Revised: We report the results in Table 3.

Original: Similar to the observationsin semi-supervised classification, the SSL methods outperform or are on par with the supervisedcounterparts in all tasks.
Revised: Similar to the observationsin semi-supervised classification, the SSL methods outperform or are on par with the supervisedcounterparts in all tasks.

Original: Our method achieves better performance than previous approaches in mostof the tasks and metrics considered.
Revised: Our method achieves better performance than previous approaches in mostof the tasks and metrics considered.

Original: In particular, MEC is significantly better than the supervisedbaseline (+1.6 AP),
Revised: In particular, MEC is significantly better than the supervisedbaseline (+1.6 AP),

Original: SimCLR (+1.9 AP), and previous best method MoCo v2 (+0.5 AP) on COCOdetection task.
Revised: SimCLR (+1.9 AP), and previous best method MoCo v2 (+0.5 AP) on COCOdetection task.

Original: These results validate that the model pre-trained by MEC can generalize very wellacross different image-based tasks and datasets.
Revised: These results validate that the model pre-trained by MEC can generalize very wellacross different image-based tasks and datasets.

Original: Video-based tasks.
Revised: Video-based tasks.

Original: To further evaluate whether MEC’s representation generalizes beyond imagebased tasks, we transfer the pre-trained model to a series of video tasks based on a recently proposedevaluation platform named UniTrack [66].
Revised: To further evaluate whether MEC’s representation generalizes beyond imagebased tasks, we transfer the pre-trained model to a series of video tasks based on a recently proposedevaluation platform named UniTrack [71].

Original: In contrast to above evaluation processes, UniTrack doesnot require any additional training and hence provides a more direct comparison between differentrepresentations.
Revised: In contrast to above evaluation processes, UniTrack doesnot require any additional training and hence provides a more direct comparison between differentrepresentations.

Original: We report the results in Table 4 and compare different methods using radar plots in
Revised: We report the results in Table 4 and compare different methods using radar plots in

Original: Figure 1.
Revised: Figure 1.

Original: As noted in UniTrack, there is no significant correlation between linear probe accuracy andtracking performance.
Revised: As noted in UniTrack, there is no significant correlation between linear probe accuracy andtracking performance.

Original: However, our method achieves strong performance across linear probing andall five video tracking tasks.
Revised: However, our method achieves strong performance across linear probing andall five video tracking tasks.

Original: Specifically, MEC achieves the highest rank on four our of five tasks, andranks among the top 2 methods on seven metrics out of all nine metrics.
Revised: Specifically, MEC achieves the highest rank on four our of five tasks, andranks among the top 2 methods on seven metrics out of all nine metrics.

Original: These results substantiatethat the learned representations of MEC are more general-purpose and versatile.
Revised: These results substantiatethat the learned representations of MEC are more general-purpose and versatile.

Original: We attribute thisimportant property of MEC to the principle of maximum entropy, the basis of our method, whichenables the model to make best use of data for unseen downstream tasks during pre-training.
Revised: We attribute thisimportant property of MEC to the principle of maximum entropy, the basis of our method, whichenables the model to make best use of data for unseen downstream tasks during pre-training.

Original: 3.4 Ablation studies
Revised: 3.4 Ablation studies

Original: Batch size and training epoch.
Revised: Batch size and training epoch.

Original: The computation and memory resources required for trainingself-supervised models are demanding.
Revised: The computation and memory resources required for trainingself-supervised models are demanding.

Original: Previous methods often need a large batch size ( e.g. , 4096in SimCLR [12]) and long training time ( e.g. , 1000 epochs in Barlow Twins [75]) to work well.
Revised: Previous methods often need a large batch size ( e.g. , 4096in SimCLR [12]) and long training time ( e.g. , 1000 epochs in Barlow Twins [81]) to work well.

Original: All methods are pre-trained with two 224 × 224 crops on ImageNet training set, and we report the results of linear probing on ImageNet validation set.

Original: [Figure] We hence test the efficiency of our method by pre-training the model with smaller training epochs(Table 5a) and batch sizes (Table 5b).
Revised: We hence test the efficiency of our method by pre-training the model with smaller training epochs(Table 5a) and batch sizes (Table 5b).

Original: The results show that our method can effectively avoid thoseprohibitive requirements.
Revised: The results show that our method can effectively avoid thoseprohibitive requirements.

Original: In particular, Our 50-epoch pre-trained model has already outperformedthose 100-epoch pre-trained models of other methods ( e.g. , SimCLR [12], BYOL [25], Barlow 
Revised: In particular, Our 50-epoch pre-trained model has already outperformedthose 100-epoch pre-trained models of other methods ( e.g. , SimCLR [12], BYOL [27], Barlow [Table] [Table]

Original: Twins [75]).
Revised: Twins [81]).

Original: Moreover, MEC is more robust to different batch sizes, and even a batch size of 128performs well (with 1.1% accuracy drop).
Revised: Moreover, MEC is more robust to different batch sizes, and even a batch size of 128performs well (with 1.1% accuracy drop).

Original: These results are noticeably different from those methodsbased on the pretext task of instance discrimination, where a large number of negative samples isrequired ( e.g. , over 4% accuracy drop of SimCLR with a batch size of 128).
Revised: These results are noticeably different from those methodsbased on the pretext task of instance discrimination, where a large number of negative samples isrequired ( e.g. , over 4% accuracy drop of SimCLR with a batch size of 128).

Original: This behavior alsoliberates MEC from those complex designs, such as memory queue [13] and online clustering [10].
Revised: This behavior alsoliberates MEC from those complex designs, such as memory queue [13] and online clustering [10].

Original: Siamese networks play a critical role in discriminative SSL, and different methods often rely ondifferent architectures to prevent mode collapse.
Revised: Siamese networks play a critical role in discriminative SSL, and different methods often rely ondifferent architectures to prevent mode collapse.

Original: We investigate the effects of different designsof Siamese networks by varying the momentum coefficient (Table 5d) and the projector network(Table 5f).
Revised: We investigate the effects of different designsof Siamese networks by varying the momentum coefficient (Table 5d) and the projector network(Table 5f).

Original: Compared to other methods [13, 14, 25, 75], MEC still works well when the Siamesenetworks are direct weight-sharing (m=0) and symmetric (sym.).
Revised: Compared to other methods [13, 14, 27, 81], MEC still works well when the Siamesenetworks are direct weight-sharing (m=0) and symmetric (sym.).

Original: In contrast, BYOL [25] requires themomentum encoder to prevent collapse (0.3% accuracy without it), and asymmetric architectures areessential for SimSiam [14] to work.
Revised: In contrast, BYOL [27] requires themomentum encoder to prevent collapse (0.3% accuracy without it), and asymmetric architectures areessential for SimSiam [14] to work.

Original: These results demonstrate that the objective of MEC naturallyavoids representational collapse without relying on special design of Siamese networks.
Revised: These results demonstrate that the objective of MEC naturallyavoids representational collapse without relying on special design of Siamese networks.

Original: We alsoobserve that the performance can be further improved with a larger projector network, consistent withthe conclusions in previous methods [75, 5].

Original: Taylor series approximation.
Revised: Taylor series approximation.

Original: The computation of log-determinant of a high-dimensional matrix canbe significantly accelerated by exploiting Taylor series approximation.
Revised: The computation of log-determinant of a high-dimensional matrix canbe significantly accelerated by exploiting Taylor series approximation.

Original: And we study two importanthyper-parameters of the process, i.e. , the order of Taylor expansion (Table 5e) and the distortionmeasure ϵ d (Table 5c).
Revised: And we study two importanthyper-parameters of the process, i.e. , the order of Taylor expansion (Table 5e) and the distortionmeasure ϵ d (Table 5c).

Original: We find that the accuracy increases with higher order of expansion, since itproduces more accurate approximation.
Revised: We find that the accuracy increases with higher order of expansion, since itproduces more accurate approximation.

Original: Besides the linear probing, we also find the COCO detectionperformance can be improved by a large margin (+1.5 AP) when the order is increased from 1 to 4.
Revised: Besides the linear probing, we also find the COCO detectionperformance can be improved by a large margin (+1.5 AP) when the order is increased from 1 to 4.

Original: Asfor the distortion hyper-parameter, on one hand, it should be large enough so that the Taylor expansionis convergent, and we empirically observe this condition is satisfied over a wide range of degree ofdistortion.
Revised: Asfor the distortion hyper-parameter, on one hand, it should be large enough so that the Taylor expansionis convergent, and we empirically observe this condition is satisfied over a wide range of degree ofdistortion.

Original: On the other hand, a good choice of ϵ d is also needed to improve the representation quality,because it compromises the uniformity and tolerance properties of representations as depicted in
Revised: On the other hand, a good choice of ϵ d is also needed to improve the representation quality,because it compromises the uniformity and tolerance properties of representations as depicted in

Original: Figure 4 and discussed in Section 2.2.
Revised: Figure 4 and discussed in Section 2.2.

Original: Generalization across different backbones and objectives.
Revised: Generalization across different backbones and objectives.

Original: Given the recent progress in Vision
Revised: Given the recent progress in Vision

Original: Transformers (ViTs) [20], it is worthwhile to examine the generalization ability of our method on both ViTs and ConvNets, though previous methods usually focus on only one architecture.
Revised: Transformers (ViTs) [20], it is worthwhile to examine the generalization ability of our method onboth ViTs and ConvNets, though previous methods usually focus on only one architecture.

Original: To this end, wemake a straightforward extension by replacing the ConvNet encoder with a ViT encoder, followingthe practice of [15].

Original: The comparisons are made in Table 5 with previous methods based on Siamesenetworks [12, 25, 10, 11, 27], and also recently proposed masked auto-encoding methods [26,78] tailored for Transformers.
Revised: Thecomparisons are made in Table 5 with previous methods based on Siamese networks [12, 27, 10, 11,29], and also recently proposed masked auto-encoding methods [28, 84] tailored for Transformers.

Original: And the results show that MEC performs equally well with both architectures.
Revised: And the results show that MEC performs equally well with both architectures.

Original: Furthermore, since MEC is designed for optimizing the representation quality, it shouldbe complementary to other methods that focus on pretext tasks.
Revised: Furthermore, since MEC is designed for optimizing the representation quality, it should be complementary to othermethods that focus on pretext tasks.

Original: To validate this hypothesis, weapply MEC as a regulation term on the representations of a contrastive method, SimCLR [12], and anon-contrastive method, SimSiam [14].
Revised: To validate this hypothesis, we apply MEC as a regulationterm on the representations of a contrastive method, SimCLR [12], and a non-contrastive method, SimSiam [14].

Original: The results in Figure 5 demonstrate that MEC improves bothlinear and kNN accuracy by a large margin ( e.g. , 2.3% linear probing accuracy for SimCLR) withnegligible computation overhead ( ∼ 1.05 × running time). 
Revised: The results in Figure 5 demonstrate that MEC improves both linear and kNN accuracyby a large margin ( e.g. , 2.3% linear probing accuracy for SimCLR) with negligible computationoverhead ( ∼ 1.05 × running time). [Figure] kNN [Figure]

Original: Related Work
Revised: SimSiam [ [Figure] [Figure] 86.1 ✓ 84.

Original:  Maximum entropy principle states that the probability distribution which best represents the currentstate of knowledge is the one with largest entropy, in the context of precisely stated prior data, whichoriginates in Laplace’s "
Revised: SimCLR [12] 84. 85.9 ✓ 87.  Related Work Maximum entropy principle states that the probability distribution which best represents the currentstate of knowledge is the one with largest entropy, in the context of precisely stated prior data, whichoriginates in Laplace’s "

Original: Principle of Insufficient Reason" and is popularized by Jaynes [34, 35].
Revised: Principle of Insufficient Reason" and is popularized by Jaynes [36, 37].

Original: Ithas a wide variety of applications in different fields [36], ranging from statistical physics [50, 43],economics [2, 53], to reinforcement learning [67, 41] and supervised learning [49, 55, 21].
Revised: Ithas a wide variety of applications in different fields [38], ranging from statistical physics [54, 47],economics [2, 57], to reinforcement learning [72, 45] and supervised learning [53, 59, 21].

Original: To thebest of our knowledge, the usage of this principle has not been explored in the context of SSL.
Revised: To thebest of our knowledge, the usage of this principle has not been explored in the context of SSL.

Original: Wenotice a closely related work MCR 2 [74] exploits the same coding length function (Equation (1)), butthe motivation behind is quite different: they do not originate from maximum entropy, but insteadbase their method on coding rate reduction.
Revised: Wenotice a closely related work MCR 2 [80] exploits the same coding length function (Equation (1)), butthe motivation behind is quite different: they do not originate from maximum entropy, but insteadbase their method on coding rate reduction.

Original: Furthermore, MCR 2 experiments on classification andclustering tasks, and is difficult to generalize to modern SSL settings due to the expensive andunstable computation.
Revised: Furthermore, MCR 2 experiments on classification andclustering tasks, and is difficult to generalize to modern SSL settings due to the expensive andunstable computation.

Original: In contrast, our method is tailored for SSL and we reformulate the codinglength function into a scalable and stable form for large-scale pre-training.
Revised: In contrast, our method is tailored for SSL and we reformulate the codinglength function into a scalable and stable form for large-scale pre-training.

Original: Another closely relatedconcept in information theory is mutual information, which has been widely used for SSL [30, 45,58, 57, 59] based on the infomax principle [38].
Revised: Another closely relatedconcept in information theory is mutual information, which has been widely used for SSL [32, 49,63, 62, 64] based on the infomax principle [41].

Original: Our method can also be viewed as an applicationof the infomax principle that maximizes the representation information, but without using mutualinformation estimators, which might result in worse transfer performance [60].
Revised: Our method can also be viewed as an applicationof the infomax principle that maximizes the representation information, but without using mutualinformation estimators, which might result in worse transfer performance [65].

Original: Siamese representation learning uses two identical neural networks to learn representations bycomparing entities, in supervised manner [8, 56, 7], or recently in self-supervised manner [12, 27,13, 14, 25, 10, 11, 75, 5] focusing on different pretext tasks.
Revised: Siamese representation learning uses two identical neural networks to learn representations bycomparing entities, in supervised manner [8, 60, 7], or recently in self-supervised manner [12, 29,13, 14, 27, 10, 11, 81, 5] focusing on different pretext tasks.

Original: The pretext tasks [76, 77, 44, 73, 19,70, 47, 18, 26, 4] are pre-designed tasks for the model to solve without relying on human-annotatedlabels, and solving pretext tasks requires the model to understand the visual concepts present inimages and thus useful representations can be learned.
Revised: The pretext tasks [82, 83, 48, 78, 19,75, 51, 18, 28, 4] are pre-designed tasks for the model to solve without relying on human-annotatedlabels, and solving pretext tasks requires the model to understand the visual concepts present inimages and thus useful representations can be learned.

Original: In this work, however, we argue that thosehandcrafted pretext tasks, like traditional handcrafted features, might introduce undesired bias into therepresentations.
Revised: In this work, however, we argue that thosehandcrafted pretext tasks, like traditional handcrafted features, might introduce undesired bias into therepresentations.

Original: So we pursue a straightforward approach to explicitly optimize on representations,in order to minimize the biases brought by the pretext task and learn generalizable representations.
Revised: So we pursue a straightforward approach to explicitly optimize on representations,in order to minimize the biases brought by the pretext task and learn generalizable representations.

Original: Our work also leverages the basic view consistency prior inherent in Siamese representation learning.
Revised: Our work also leverages the basic view consistency prior inherent in Siamese representation learning.

Original: Compared to previous work, our method of maximum entropy coding naturally avoids the low-entropytrivial solution and can be easily implemented without requiring instance discrimination [19, 12],memory queue [70, 27, 13] or online clustering [9, 3, 10].
Revised: Compared to previous work, our method of maximum entropy coding naturally avoids the low-entropymode collapse problem and can be easily implemented without requiring instance discrimination [19,12], memory queue [75, 29, 13] or online clustering [9, 3, 10].

Original:  Conclusion
Revised:  Conclusion

Original: What makes for generalizable representations?
Revised: What makes for generalizable representations?

Original: To address this fundamental problem, the proposed
Revised: To address this fundamental problem, the proposed

Original: MEC exploits the principle of maximum entropy and incorporates the view consistency prior of
Revised: MEC exploits the principle of maximum entropy and incorporates the view consistency prior of

Original: Siamese representation learning.
Revised: Siamese representation learning.

Original: In order to facilitate large-scale pre-training, we reformulatethe coding length function into a scalable form and make the large-scale pre-training possible.
Revised: In order to facilitate large-scale pre-training, we reformulatethe coding length function into a scalable form and make the large-scale pre-training possible.

Original: And with the reformulation, we further demonstrate that MEC bridges the batch-wise and featurewise objectives.
Revised: And with the reformulation, we further demonstrate that MEC bridges the batch-wise and featurewise objectives.

Original: Moreover, we show MEC generalizes well across different architectures, andis complementary to existing methods and further improves them when integrated.
Revised: Moreover, we show MEC generalizes well across different architectures, andis complementary to existing methods and further improves them when integrated.

Original: Extensiveexperiments on various image- and video-based downstream tasks demonstrate that the learnedrepresentations by MEC are more general-purpose and versatile.
Revised: Extensiveexperiments on various image- and video-based downstream tasks demonstrate that the learnedrepresentations by MEC are more general-purpose and versatile.

Original: We hope our exploration will inspirepeople to rethink the fundamental problem in representation learning and design more methods.
Revised: We hope our exploration will inspirepeople to rethink the fundamental problem in representation learning and design more methods.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Please see Appendix E.
Revised: [Yes] Please see Appendix E.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Pleasesee Appendix F.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] Please see Appendix F.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]3.
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] A pseudocode is
Revised: [Yes] A pseudocode is

Original: provided in Appendix A and code will be released.
Revised: provided in Appendix A and code will be released.

Original: All instructions are in Appendix C.
Revised: All instructions are in Appendix C.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] All implementation details are described in Appendix C.
Revised: [Yes] All implementation details are described in Appendix C.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] We report the average performance over three seeds.
Revised: [Yes] We report the average performance over three seeds.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Please see Appendix C.
Revised: [Yes] Please see Appendix C.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Please see Appendix G.
Revised: [Yes] Please see Appendix G.

Original: (c) Did you include any new assets either in the supplemental or as a URL?
Revised: (c) Did you include any new assets either in the supplemental or as a URL?

Original: [N/ A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?[N/A]
Revised: using/curating?[N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]