Original: Self-Supervised Learning via Maximum Entropy Coding
Revised: Self-Supervised Learning via Maximum Entropy Coding

Original: Abstract
Revised: Abstract

Original: A mainstream type of current self-supervised learning methods pursues a generalpurpose representation that can be well transferred to downstream tasks, typically by optimizing on a given pretext task such as instance discrimination.
Revised: A mainstream type of current self-supervised learning methods pursues a generalpurpose representation that can be well transferred to downstream tasks, typically by optimizing on a given pretext task such as instance discrimination.

Original: In this work, we argue that existing pretext tasks inevitably introduce biases into the learned representation, which in turn leads to biased transfer performance on various downstream tasks.
Revised: In this work, we argue that existing pretext tasks inevitably introduce biases into the learned representation, which in turn leads to biased transfer performance on various downstream tasks.

Original: To cope with this issue, we propose Maximum Entropy Coding ( MEC ), a more principled objective that explicitly optimizes on the structure of the representation, so that the learned representation is less biased and thus generalizes better to unseen downstream tasks.
Revised: To cope with this issue, we propose Maximum Entropy Coding ( MEC ), a more principled objective that explicitly optimizes on the structure of the representation, so that the learned representation is less biased and thus generalizes better to unseen downstream tasks.

Original: Inspired by the principle of maximum entropy in information theory, we hypothesize that a generalizable representation should be the one that admits the maximum entropy among all plausible representations.
Revised: Inspired by the principle of maximum entropy in information theory, we hypothesize that a generalizable representation should be the one that admits the maximum entropy among all plausible representations.

Original: To make the objective end-to-end trainable, we propose to leverage the minimal coding length in lossy data coding as a computationally tractable surrogate for the entropy, and further derive a scalable reformulation of the objective that allows fast computation.
Revised: To make the objective end-to-end trainable, we propose to leverage the minimal coding length in lossy data coding as a computationally tractable surrogate for the entropy, and further derive a scalable reformulation of the objective that allows fast computation.

Original: Extensive experiments demonstrate that MEC learns a more generalizable representation than previous methods based on specific pretext tasks.
Revised: Extensive experiments demonstrate that MEC learns a more generalizable representation than previous methods based on specific pretext tasks.

Original: It achieves state-of-the-art performance consistently on various downstream tasks, including not only ImageNet linear probe, but also semi-supervised classification, object detection, instance segmentation, and object tracking.
Revised: It achieves state-of-the-art performance consistently on various downstream tasks, including not only ImageNet linear probe, but also semi-supervised classification, object detection, instance segmentation, and object tracking.

Original: Interestingly, we show that existing batch-wise and feature-wise self-supervised objectives could be seen equivalent to low-order approximations of MEC.
Revised: Interestingly, we show that existing batch-wise and feature-wise self-supervised objectives could be seen equivalent to low-order approximations of MEC.

Original: Code and pre-trained models are available at https://github.com/xinliu20/MEC .
Revised: Code and pre-trained models are available at https://github.com/xinliu20/MEC .

Original:  Introduction
Revised:  Introduction

Original: Self-supervised learning (SSL) aims to learn rich and meaningful representations without relying on human annotations.
Revised: Self-supervised learning (SSL) aims to learn rich and meaningful representations without relying on human annotations.

Original: Pursuing general-purpose representations, SSL models are typically used as pre-trained weights for providing a good initialization to downstream tasks.
Revised: Pursuing general-purpose representations, SSL models are typically used as pre-trained weights for providing a good initialization to downstream tasks.

Original: In this sense, SSL [12, 31, 10, 29, 15, 14, 86, 30] has seen great progress in computer vision, and can achieve competitive or even better performance on various downstream tasks compared to its supervised counterparts.
Revised: In this sense, SSL [12, 31, 10, 29, 15, 14, 86, 30] has seen great progress in computer vision, and can achieve competitive or even better performance on various downstream tasks compared to its supervised counterparts.

Original: At the core of current SSL methods is the design of pretext tasks .
Revised: At the core of current SSL methods is the design of pretext tasks .

Original: A pretext task is a (usually hand-crafted) learning objective in which the supervision signals could be mined from the data itself, with the aim of applying the learned representation to other downstream tasks.
Revised: A pretext task is a (usually hand-crafted) learning objective in which the supervision signals could be mined from the data itself, with the aim of applying the learned representation to other downstream tasks.

Original: Early attempts of pretext tasks typically discard a certain property of the input data and then force the model to predict the discarded property.
Revised: Early attempts of pretext tasks typically discard a certain property of the input data and then force the model to predict the discarded property.

Original: For instance, one can convert an RGB image to a gray-scale one, and train the model to predict the original color [87, 88]; or apply random rotation to image patches and ask the
Revised: For instance, one can convert an RGB image to a gray-scale one, and train the model to predict the original color [87, 88]; or apply random rotation to image patches and ask the

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: SimCLR
Revised: SimCLR

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: MoCo-v
Revised: MoCo-v

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: BYOL
Revised: BYOL

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: SwAV
Revised: SwAV

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: SimSiam
Revised: SimSiam

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: BarlowTwins
Revised: BarlowTwins

Original: LIN
Revised: LIN

Original: S E M
Revised: S E M

Original: I
Revised: I

Original: V O C C O C O
Revised: V O C C O C O

Original: S E G
Revised: S E G

Original: MEC
Revised: MEC

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: InsDis
Revised: InsDis

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: SimCLR
Revised: SimCLR

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: MoCo-v
Revised: MoCo-v

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: InfoMin
Revised: InfoMin

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: BYOL
Revised: BYOL

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: SwAV
Revised: SwAV

Original: SOT
Revised: SOT

Original: V O S
Revised: V O S

Original: M O T P o s e T
Revised: M O T P o s e T

Original: M O T S
Revised: M O T S

Original: MEC model to predict the rotation angles [27].
Revised: MEC model to predict the rotation angles [27].

Original: In both cases, to fulfill the pretext task, the model must learn a meaningful representation that can describe object texture, shape or even category, therefore the learned representation transfers well to downstream tasks related to these features.
Revised: In both cases, to fulfill the pretext task, the model must learn a meaningful representation that can describe object texture, shape or even category, therefore the learned representation transfers well to downstream tasks related to these features.

Original: Despite successes of existing pretext tasks, we find that they inevitably introduce biases into the learned representation, which conflicts with the original aim of “general-purpose”.
Revised: Despite successes of existing pretext tasks, we find that they inevitably introduce biases into the learned representation, which conflicts with the original aim of “general-purpose”.

Original: For example, representations trained with the most prevalent image-level task, instance discrimination [20, 80], are found biased to image-level tasks such as image classification, while by contrast degenerate in patchor pixel-level tasks like object detection and semantic segmentation [81].
Revised: For example, representations trained with the most prevalent image-level task, instance discrimination [20, 80], are found biased to image-level tasks such as image classification, while by contrast degenerate in patchor pixel-level tasks like object detection and semantic segmentation [81].

Original: Even being transferred to image-level tasks, such representations still suffer from domain gaps in cases of different data distributions [26], e.g. , classification on unseen categories other than ImageNet objects.
Revised: Even being transferred to image-level tasks, such representations still suffer from domain gaps in cases of different data distributions [26], e.g. , classification on unseen categories other than ImageNet objects.

Original: In this work, we are curious of what makes for generalizable representations , and pursue an explicit optimization with a criterion that directly measures the structure of representations, with the aim of minimizing the biases brought by the pretext task.
Revised: In this work, we are curious of what makes for generalizable representations , and pursue an explicit optimization with a criterion that directly measures the structure of representations, with the aim of minimizing the biases brought by the pretext task.

Original: To this end, we propose Maximum Entropy
Revised: To this end, we propose Maximum Entropy

Original: Coding (MEC).
Revised: Coding (MEC).

Original: Inspired by the principle of maximum entropy in information theory, the basic hypothesis in MEC is that a generalizable representation should be the one that admit the maximum entropy among all plausible representations.
Revised: Inspired by the principle of maximum entropy in information theory, the basic hypothesis in MEC is that a generalizable representation should be the one that admit the maximum entropy among all plausible representations.

Original: Accordingly, optimizing towards maximum entropy leads to representations with good generalization capacity.
Revised: Accordingly, optimizing towards maximum entropy leads to representations with good generalization capacity.

Original: The main challenge confronting us is that it is difficult, and computationally expensive, if possible, to estimate the distribution of a given representation (usually a finite set of high-dimensional vectors), so in turn it is difficult to estimate the entropy.
Revised: The main challenge confronting us is that it is difficult, and computationally expensive, if possible, to estimate the distribution of a given representation (usually a finite set of high-dimensional vectors), so in turn it is difficult to estimate the entropy.

Original: To cope with this issue, we replace the optimization objective from the originally defined entropy to a computationally tractable surrogate, i.e. , the necessary number of bits needed to encode the representations via lossy data coding [16].
Revised: To cope with this issue, we replace the optimization objective from the originally defined entropy to a computationally tractable surrogate, i.e. , the necessary number of bits needed to encode the representations via lossy data coding [16].

Original: The log-determinant term costs the most computation in the coding length function.
Revised: The log-determinant term costs the most computation in the coding length function.

Original: By leveraging Taylor expansion of matrix, we further approximate this term with polynomial functions, leading to significant speed up and thus makes large-scale pre-training possible.
Revised: By leveraging Taylor expansion of matrix, we further approximate this term with polynomial functions, leading to significant speed up and thus makes large-scale pre-training possible.

Original: In contrast to previous SSL methods that mainly evaluate on ImageNet classification, we experiment on a wide variety of vision tasks to show the good generalization capacity of MEC.
Revised: In contrast to previous SSL methods that mainly evaluate on ImageNet classification, we experiment on a wide variety of vision tasks to show the good generalization capacity of MEC.

Original: The considered tasks span across not only image-level recognition on various data distributions, but also patch- or pixel-level tasks like object detection, instance segmentation, and object tracking.
Revised: The considered tasks span across not only image-level recognition on various data distributions, but also patch- or pixel-level tasks like object detection, instance segmentation, and object tracking.

Original: We show MEC generalizes well consistently across all tasks considered, while representations learned with previous pretext tasks usually perform well on closely related downstream tasks but degenerate on less related ones (see Figure 1).
Revised: We show MEC generalizes well consistently across all tasks considered, while representations learned with previous pretext tasks usually perform well on closely related downstream tasks but degenerate on less related ones (see Figure 1).

Original: Besides empirical results, we find interesting equivalence between low-order approximations of MEC and
Revised: Besides empirical results, we find interesting equivalence between low-order approximations of MEC and

Original: existing batch-wise ( e.g. , SimSiam [14]) or feature-wise objectives ( e.g. ,
Revised: existing batch-wise ( e.g. , SimSiam [14]) or feature-wise objectives ( e.g. ,

Original: Barlow Twins [86]), which provides a new perspective for a unified understanding of prevalent SSL methods.
Revised: Barlow Twins [86]), which provides a new perspective for a unified understanding of prevalent SSL methods.

Original: In summary, the main contributions of our work are as follows:
Revised: In summary, the main contributions of our work are as follows:

Original: • To learn representations generalizable to various downstream tasks, we introduce the principle of maximum entropy into self-supervised learning.
Revised: • To learn representations generalizable to various downstream tasks, we introduce the principle of maximum entropy into self-supervised learning.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /AoLAwix8g9s/A1OmwFajnSlo3Pu1b33eBGjUlnWt1FYW9/Y3Cpul3Z29/YPzMOjjgxjgUkbhywUPQ9JwignbUUVI71IEBR4jHS9yU3mdx+IkDTkLTWLiBOgEac+xUhpyTVPK4MAqTFGLLlL3aSBBAunsDWlXKYV1yxbVWsOuErsnJRBjqZrfg2GIY4DwhVmSMq+bUXKSZBQFDOSlgaxJBHCEzQifU05Coh0kvkXKTzXyhD6odDFFZyrvycSFEg5CzzdmZ0sl71M/M/rx8q/dhLKo1gRjheL/JhBFcIsEjikgmDFZpogLKi+FeIxEggrHVxJh2Avv7xKOrWqfVm172vleiOPowhOwBm4ADa4AnVwC5qgDTB4BM/gFbwZT8aL8W58LFoLRj5zDP7A
Revised: /AoLAwix8g9s/A1OmwFajnSlo3Pu1b33eBGjUlnWt1FYW9/Y3Cpul3Z29/YPzMOjjgxjgUkbhywUPQ9JwignbUUVI71IEBR4jHS9yU3mdx+IkDTkLTWLiBOgEac+xUhpyTVPK4MAqTFGLLlL3aSBBAunsDWlXKYV1yxbVWsOuErsnJRBjqZrfg2GIY4DwhVmSMq+bUXKSZBQFDOSlgaxJBHCEzQifU05Coh0kvkXKTzXyhD6odDFFZyrvycSFEg5CzzdmZ0sl71M/M/rx8q/dhLKo1gRjheL/JhBFcIsEjikgmDFZpogLKi+FeIxEggrHVxJh2Avv7xKOrWqfVm172vleiOPowhOwBm4ADa4AnVwC5qgDTB4BM/gFbwZT8aL8W58LFoLRj5zDP7A

Original: [Figure]
Revised: [Figure]

Original: PwBXTKYfA
Revised: PwBXTKYfA

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: L
Revised: L

Original: [Figure]
Revised: [Figure]

Original: • We propose Maximum Entropy Coding ( MEC ), which explicitly optimizes on representations based on the principle of maximum entropy, and leverages the minimal coding length in lossy data coding as a computationally tractable surrogate for the entropy.
Revised: • We propose Maximum Entropy Coding ( MEC ), which explicitly optimizes on representations based on the principle of maximum entropy, and leverages the minimal coding length in lossy data coding as a computationally tractable surrogate for the entropy.

Original: • We reformulate the log-determinant term in coding length function into a scalable form, which makes large-scale pre-training possible, and unifies existing batch-wise and feature-wise objectives as low-order approximations of our method.
Revised: • We reformulate the log-determinant term in coding length function into a scalable form, which makes large-scale pre-training possible, and unifies existing batch-wise and feature-wise objectives as low-order approximations of our method.

Original: • We show MEC representation generalizes well on a wide variety of image- and video-based downstream tasks, achieving state-of-the-arts on most tasks considered.
Revised: • We show MEC representation generalizes well on a wide variety of image- and video-based downstream tasks, achieving state-of-the-arts on most tasks considered.

Original:  Method
Revised:  Method

Original: In this section, we start with illustrating the maximum entropy principle, and then introduce a computationally tractable surrogate of the information entropy for high-dimensional vectors.
Revised: In this section, we start with illustrating the maximum entropy principle, and then introduce a computationally tractable surrogate of the information entropy for high-dimensional vectors.

Original: We then present a scalable reformulation of the proposed surrogate, which makes large-scale training possible.
Revised: We then present a scalable reformulation of the proposed surrogate, which makes large-scale training possible.

Original: And we further incorporate the view consistency prior for maximum entropy coding.
Revised: And we further incorporate the view consistency prior for maximum entropy coding.

Original: Finally, we demonstrate how the proposed method can unify existing batch-wise and feature-wise SSL objectives.
Revised: Finally, we demonstrate how the proposed method can unify existing batch-wise and feature-wise SSL objectives.

Original: Please refer to Appendix E for more details about the proofs in this section.
Revised: Please refer to Appendix E for more details about the proofs in this section.

Original: 2.1 Maximum Entropy Coding
Revised: 2.1 Maximum Entropy Coding

Original: The maximum entropy principle.
Revised: The maximum entropy principle.

Original: The main purpose of this work is to improve the generalization capacity of self-supervised learning representations across unseen downstream tasks and data distributions, and reduce the biases brought by specifically designed pretext tasks as much as possible.
Revised: The main purpose of this work is to improve the generalization capacity of self-supervised learning representations across unseen downstream tasks and data distributions, and reduce the biases brought by specifically designed pretext tasks as much as possible.

Original: This naturally raises a question, i.e. , what makes for a generalizable representation?
Revised: This naturally raises a question, i.e. , what makes for a generalizable representation?

Original: To answer this question, we are particularly inspired by the maximum entropy principle in information theory, which states that the probability distribution that best represents the current state of knowledge about a system is the one with largest entropy, given a testable information (such as accuracy) and in this way no additional bias or assumptions is introduced [38, 39, 59].
Revised: To answer this question, we are particularly inspired by the maximum entropy principle in information theory, which states that the probability distribution that best represents the current state of knowledge about a system is the one with largest entropy, given a testable information (such as accuracy) and in this way no additional bias or assumptions is introduced [38, 39, 59].

Original: We therefore hypothesis that a generalizable representation is the one that has the maximum entropy among all plausible representations.
Revised: We therefore hypothesis that a generalizable representation is the one that has the maximum entropy among all plausible representations.

Original: Intuitively, if we are able to express the entropy in a closed form, the maximum entropy principle then can serve as an optimization objective and supervise the representation learning.
Revised: Intuitively, if we are able to express the entropy in a closed form, the maximum entropy principle then can serve as an optimization objective and supervise the representation learning.

Original: Minimal coding length as a surrogate for entropy.
Revised: Minimal coding length as a surrogate for entropy.

Original: Entropy is originally defined on probability distributions [63], i.e. , H ( z ) ≜ − (cid:82) p ( z ) log p ( z ) dz , for continuous random variables.
Revised: Entropy is originally defined on probability distributions [63], i.e. , H ( z ) ≜ − (cid:82) p ( z ) log p ( z ) dz , for continuous random variables.

Original: However, it is very difficult to estimate the true distributions p ( z ) of a representation [6, 54], from a finite set of high dimensional vectors Z = (cid:2)
Revised: However, it is very difficult to estimate the true distributions p ( z ) of a representation [6, 54], from a finite set of high dimensional vectors Z = (cid:2)

Original: z 1 , z 2 , . .
Revised: z 1 , z 2 , . .

Original: , z m (cid:3) ∈ R d × m .
Revised: , z m (cid:3) ∈ R d × m .

Original: A handy fact is that entropy is conceptually equivalent to the minimal number of bits required to encode the data losslessly, so the minimal lossless coding length could be used to represent the entropy.
Revised: A handy fact is that entropy is conceptually equivalent to the minimal number of bits required to encode the data losslessly, so the minimal lossless coding length could be used to represent the entropy.

Original: However, lossless coding of continuous random variables is infeasible in our case since it often requires an infinite number of bits, breaking the numerical stability.
Revised: However, lossless coding of continuous random variables is infeasible in our case since it often requires an infinite number of bits, breaking the numerical stability.

Original: Instead, we exploit the coding length in lossy data coding [16] as a computationally tractable surrogate for the entropy of continuous random variables.
Revised: Instead, we exploit the coding length in lossy data coding [16] as a computationally tractable surrogate for the entropy of continuous random variables.

Original: Given a set of samples Z , the minimal number of bits needed to encode Z subject to a distortion ϵ is given by the
Revised: Given a set of samples Z , the minimal number of bits needed to encode Z subject to a distortion ϵ is given by the

Original: [Figure]
Revised: [Figure]

Original: following coding length function [47, 72]:
Revised: following coding length function [47, 72]:

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: where I m denotes the identity matrix with dimension m , and ϵ is the upper bound of the expected decoding error between z ∈ Z and the decoded (cid:98) z , i.e. , E [ ∥ z − (cid:98)
Revised: where I m denotes the identity matrix with dimension m , and ϵ is the upper bound of the expected decoding error between z ∈ Z and the decoded (cid:98) z , i.e. , E [ ∥ z − (cid:98)

Original: z
Revised: z

Original: ∥ 2 ] ≤ ϵ .
Revised: ∥ 2 ] ≤ ϵ .

Original: We note that the computation of log-determinant of high dimensional matrix in Equation (1) is highly expensive and may cause numerically unstable results for ill-conditioned matrix, which inhibits its application to large-scale pre-training ( e.g. , over 1 million images).
Revised: We note that the computation of log-determinant of high dimensional matrix in Equation (1) is highly expensive and may cause numerically unstable results for ill-conditioned matrix, which inhibits its application to large-scale pre-training ( e.g. , over 1 million images).

Original: Therefore, a scalable and stable reformulation of Equation (1) is required.
Revised: Therefore, a scalable and stable reformulation of Equation (1) is required.

Original: We first rewrite Equation (1) as L = µ log det (cid:16) I m + λ Z ⊤ Z (cid:17) , where µ = m + dand λ = dmϵ 2 .
Revised: We first rewrite Equation (1) as L = µ log det (cid:16) I m + λ Z ⊤ Z (cid:17) , where µ = m + dand λ = dmϵ 2 .

Original: Utilizing the identical equation det(exp( A ))
Revised: Utilizing the identical equation det(exp( A ))

Original: =
Revised: =

Original: exp(Tr( A )) [35], we obtain L =
Revised: exp(Tr( A )) [35], we obtain L =

Original: Tr (cid:16) µ log (cid:16) I m + λ Z ⊤ Z (cid:17)(cid:17) , where Tr stands for the trace of the matrix.
Revised: Tr (cid:16) µ log (cid:16) I m + λ Z ⊤ Z (cid:17)(cid:17) , where Tr stands for the trace of the matrix.

Original: Finally, we apply Taylor series expansion to expand the logarithm of the matrix and obtain
Revised: Finally, we apply Taylor series expansion to expand the logarithm of the matrix and obtain

Original: L =
Revised: L =

Original: Tr
Revised: Tr

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: (1) (origin) and Equation (2) (approx.)
Revised: (1) (origin) and Equation (2) (approx.)

Original: for different number of samples in Z (dim).
Revised: for different number of samples in Z (dim).

Original: with convergence condition: (cid:13)(cid:13)(cid:13) λ Z ⊤ Z (cid:13)(cid:13)(cid:13) 2 < 1 , and this can be achieved by adjusting the hyperparameter ϵ (detailed in Section 2.2).
Revised: with convergence condition: (cid:13)(cid:13)(cid:13) λ Z ⊤ Z (cid:13)(cid:13)(cid:13) 2 < 1 , and this can be achieved by adjusting the hyperparameter ϵ (detailed in Section 2.2).

Original: Compared to Equation (1), there are only matrix multiplication and addition in Equation (2), which significantly speeds up the computation process and avoids the numerical unstability.
Revised: Compared to Equation (1), there are only matrix multiplication and addition in Equation (2), which significantly speeds up the computation process and avoids the numerical unstability.

Original: To verify this, in Figure 3, we make a comparison of running time and relative approximation error between Equation (1) and (2) (using the first four terms).
Revised: To verify this, in Figure 3, we make a comparison of running time and relative approximation error between Equation (1) and (2) (using the first four terms).

Original: The results show that our reformulation can approximate the original coding length function with negligible error (all errors are well below 0 .
Revised: The results show that our reformulation can approximate the original coding length function with negligible error (all errors are well below 0 .

Original: and accelerate the computation considerably (over 50 × acceleration for all cases), thus making large-scale pre-training possible.
Revised: and accelerate the computation considerably (over 50 × acceleration for all cases), thus making large-scale pre-training possible.

Original: 2.2 Combining with the View Consistency Prior
Revised: 2.2 Combining with the View Consistency Prior

Original: It should be noted that the necessary premise of the maximum entropy principle is that testable information is given as prior.
Revised: It should be noted that the necessary premise of the maximum entropy principle is that testable information is given as prior.

Original: For example, the testable information could be the accuracy of a predictive model: the most generalizable model should be the one with maximum entropy, but it is only when a group of models reaches a given accuracy.
Revised: For example, the testable information could be the accuracy of a predictive model: the most generalizable model should be the one with maximum entropy, but it is only when a group of models reaches a given accuracy.

Original: Otherwise, simply optimizing towards maximum entropy will lead to trivial solutions such as uniform distributions.
Revised: Otherwise, simply optimizing towards maximum entropy will lead to trivial solutions such as uniform distributions.

Original: In Equation (1), the prior is not considered.
Revised: In Equation (1), the prior is not considered.

Original: To introduce a prior, we employ a common practice in SSL by augmenting Z into two different views Z 1 and Z 2 .
Revised: To introduce a prior, we employ a common practice in SSL by augmenting Z into two different views Z 1 and Z 2 .

Original: More specifically, given a set of images D , an image x is sampled uniformly from D , and two augmented views x 1 and x 2 are obtained from x via a distribution of data augmentations
Revised: More specifically, given a set of images D , an image x is sampled uniformly from D , and two augmented views x 1 and x 2 are obtained from x via a distribution of data augmentations

Original: T .
Revised: T .

Original: Then they are fed to an encoder f , consisting of a backbone and a projector network, which produces ℓ 2 -normalized embeddings of zand z 2 .
Revised: Then they are fed to an encoder f , consisting of a backbone and a projector network, which produces ℓ 2 -normalized embeddings of zand z 2 .

Original: For a batch of m images, we have Z 1 = (cid:2)
Revised: For a batch of m images, we have Z 1 = (cid:2)

Original: z 11 , z 21 , . .
Revised: z 11 , z 21 , . .

Original: z m 1 (cid:3) and similarly for Z 2 , which are two observations of the same Z .
Revised: z m 1 (cid:3) and similarly for Z 2 , which are two observations of the same Z .

Original: MEC aims to minimize the following loss:
Revised: MEC aims to minimize the following loss:

Original: [Equation]
Revised: [Equation]

Original: where the same notations in Section 2.1 apply, and n is the order of Taylor expansion.
Revised: where the same notations in Section 2.1 apply, and n is the order of Taylor expansion.

Original: Compared with Equation (1), the formulation in (3) considers not only maximizing entropy, but also the view consistency prior mined from the data itself, therefore learning meaningful representations.
Revised: Compared with Equation (1), the formulation in (3) considers not only maximizing entropy, but also the view consistency prior mined from the data itself, therefore learning meaningful representations.

Original: As noted in Section 2.1, the convergence condition of Taylor expansion requires ∥ C ∥ 2 < 1 , where
Revised: As noted in Section 2.1, the convergence condition of Taylor expansion requires ∥ C ∥ 2 < 1 , where

Original: C = λ
Revised: C = λ

Original: Z ⊤ 1 Z 2 and λ = dmϵ 2 = 1 mϵ 2 d .
Revised: Z ⊤ 1 Z 2 and λ = dmϵ 2 = 1 mϵ 2 d .

Original: We show such condition can be strictly satisfied by setting ϵ 2 d > 1 because of the inequality
Revised: We show such condition can be strictly satisfied by setting ϵ 2 d > 1 because of the inequality

Original: ∥ C ∥ 2 ≤ (cid:112) ∥ C ∥ 1 ∥ C ∥ ∞ < 1 .
Revised: ∥ C ∥ 2 ≤ (cid:112) ∥ C ∥ 1 ∥ C ∥ ∞ < 1 .

Original: In practice, we empirically find that the Taylor expansion converges over a wide range of ϵ d (Figure 4(c)) with a linear warm-up.
Revised: In practice, we empirically find that the Taylor expansion converges over a wide range of ϵ d (Figure 4(c)) with a linear warm-up.

Original: From the preliminary experiments on CIFAR-10 [41] (detailed in Appendix B), we also find that the distributions of representations show progressive finer granularity as ϵ d decreases (Figure 4(b)).
Revised: From the preliminary experiments on CIFAR-10 [41] (detailed in Appendix B), we also find that the distributions of representations show progressive finer granularity as ϵ d decreases (Figure 4(b)).

Original: This can be interpreted by the practical meaning of the distortion ϵ d (Figure 4(a)), i.e. , a smaller ϵ d encourages the representation space to be encoded in finer granularity (and hence more uniform).
Revised: This can be interpreted by the practical meaning of the distortion ϵ d (Figure 4(a)), i.e. , a smaller ϵ d encourages the representation space to be encoded in finer granularity (and hence more uniform).

Original: By contrast, a small ϵ d might break the semantic structures of similar images ( i.e. , tolerance).
Revised: By contrast, a small ϵ d might break the semantic structures of similar images ( i.e. , tolerance).

Original: Therefore, a good choice of ϵ d is needed to compromise the uniformity and tolerance properties [75, 74] of representations, which shares the same role as the temperature [80] term in contrastive learning.
Revised: Therefore, a good choice of ϵ d is needed to compromise the uniformity and tolerance properties [75, 74] of representations, which shares the same role as the temperature [80] term in contrastive learning.

Original: An overview of MEC is illustrated in Figure 2 and a PyTorch-like pseudocode is provided in
Revised: An overview of MEC is illustrated in Figure 2 and a PyTorch-like pseudocode is provided in

Original: Appendix A.
Revised: Appendix A.

Original: The algorithm describes the minimalist variant of MEC, which can be further improved by integrating momentum encoder and asymmetric networks (detailed in experiments).
Revised: The algorithm describes the minimalist variant of MEC, which can be further improved by integrating momentum encoder and asymmetric networks (detailed in experiments).

Original: 2.3 A Unified View of Batch-wise and Feature-wise SSL Objectives
Revised: 2.3 A Unified View of Batch-wise and Feature-wise SSL Objectives

Original: Current SSL methods based on Siamese networks can be roughly divided into two categories: batchwise methods [12, 31, 13, 15, 14, 10] and feature-wise methods [86, 5, 24, 36].
Revised: Current SSL methods based on Siamese networks can be roughly divided into two categories: batchwise methods [12, 31, 13, 15, 14, 10] and feature-wise methods [86, 5, 24, 36].

Original: The former aims to minimize the distance between augmented views of the same sample while maximizing the distance between different samples, which can be viewed as decorrelating the different features in a batch.
Revised: The former aims to minimize the distance between augmented views of the same sample while maximizing the distance between different samples, which can be viewed as decorrelating the different features in a batch.

Original: The latter, in contrast, tries to decorrelate the different vector components in the representation.
Revised: The latter, in contrast, tries to decorrelate the different vector components in the representation.

Original: The relationship between them has not been fully understood.
Revised: The relationship between them has not been fully understood.

Original: Our work builds bridges between these two types of methods through the following derivation:
Revised: Our work builds bridges between these two types of methods through the following derivation:

Original: [Equation]
Revised: [Equation]

Original: which can be proved since Z ⊤ 1 Z 2 ∈ R m × m and Z 1 Z ⊤
Revised: which can be proved since Z ⊤ 1 Z 2 ∈ R m × m and Z 1 Z ⊤

Original: 2 ∈ R
Revised: 2 ∈ R

Original: d × d have the same nonzero eigenvalues.
Revised: d × d have the same nonzero eigenvalues.

Original: In Figure 2, under the framework of MEC, we show the equivalence between batch-wise and featurewise methods using two examples, SimSiam [14] and Barlow Twins [86].
Revised: In Figure 2, under the framework of MEC, we show the equivalence between batch-wise and featurewise methods using two examples, SimSiam [14] and Barlow Twins [86].

Original: By taking Taylor expansion (Equation (2)) of the left side of Equation (4) and before the trace operation, the diagonal elements of the leading term ( i.e. , µλ Z ⊤ 1 Z 2 ) measure the similarity between the views of the same images in a batch, and the objective of SimSiam [14] is equivalent to maximizing the trace of this term.
Revised: By taking Taylor expansion (Equation (2)) of the left side of Equation (4) and before the trace operation, the diagonal elements of the leading term ( i.e. , µλ Z ⊤ 1 Z 2 ) measure the similarity between the views of the same images in a batch, and the objective of SimSiam [14] is equivalent to maximizing the trace of this term.

Original: Similarly, the leading term of the right side expansion models the correlation between dimensions of the feature, and the objective of Barlow Twins [86] is equivalent to the second-order expansion of L MEC .
Revised: Similarly, the leading term of the right side expansion models the correlation between dimensions of the feature, and the objective of Barlow Twins [86] is equivalent to the second-order expansion of L MEC .

Original: With the above derivation, our method naturally subsumes the two different kinds of objectives as its low-order expansions, and we show in experiments that better downstream task performance can be achieved with higher-order approximations.
Revised: With the above derivation, our method naturally subsumes the two different kinds of objectives as its low-order expansions, and we show in experiments that better downstream task performance can be achieved with higher-order approximations.

Original: We further show in Appendix E that our MEC can also bridge other self-supervised objectives.
Revised: We further show in Appendix E that our MEC can also bridge other self-supervised objectives.

Original: And we hope the direct tying of a family of objectives to a very grounded mathematical concept can inspire more new methods.
Revised: And we hope the direct tying of a family of objectives to a very grounded mathematical concept can inspire more new methods.

Original:  Experiments
Revised:  Experiments

Original: We perform self-supervised pre-training using the proposed MEC on the training set of the ImageNet ILSVRC-2012 dataset [17].
Revised: We perform self-supervised pre-training using the proposed MEC on the training set of the ImageNet ILSVRC-2012 dataset [17].

Original: After pre-training, we conduct extensive experiments to examine
Revised: After pre-training, we conduct extensive experiments to examine

Original: [Table] [Table]
Revised: [Table] [Table]

Original: VOC detection tasks and Mask R-CNN [32] (1 × schedule) for COCO detection and instance segmentation tasks.
Revised: VOC detection tasks and Mask R-CNN [32] (1 × schedule) for COCO detection and instance segmentation tasks.

Original: All Faster/Mask R-CNN models are with the C4-backbone [79].
Revised: All Faster/Mask R-CNN models are with the C4-backbone [79].

Original: Bold entries are within 0.2 below the best.
Revised: Bold entries are within 0.2 below the best.

Original: [Table]
Revised: [Table]

Original: the learned representations on various tasks.
Revised: the learned representations on various tasks.

Original: The tasks considered include four image-level tasks:
Revised: The tasks considered include four image-level tasks:

Original: ImageNet linear probing, ImageNet semi-supervised classification, object detection, and instance segmentation; and five video-based tasks: single object tracking (SOT) [78], video object segmentation (VOS) [56], multi-object detection (MOT) [48], multi-object detection and segmentation (MOTS) [73] and pose tracking (PoseTrack) [1].
Revised: ImageNet linear probing, ImageNet semi-supervised classification, object detection, and instance segmentation; and five video-based tasks: single object tracking (SOT) [78], video object segmentation (VOS) [56], multi-object detection (MOT) [48], multi-object detection and segmentation (MOTS) [73] and pose tracking (PoseTrack) [1].

Original: An overview comparison with existing SSL representations across different tasks is presented in Section 3.1.
Revised: An overview comparison with existing SSL representations across different tasks is presented in Section 3.1.

Original: Moreover, in Section 3.4, we present extensive ablations studies to demonstrate the behaviors and properties of the proposed method.
Revised: Moreover, in Section 3.4, we present extensive ablations studies to demonstrate the behaviors and properties of the proposed method.

Original: All the implementation details of the aforementioned experiments are deferred to Appendix C.
Revised: All the implementation details of the aforementioned experiments are deferred to Appendix C.

Original: 3.1 Overview Comparison
Revised: 3.1 Overview Comparison

Original: We evaluate MEC together with multiple previous SSL representations on the four image-based tasks and the five video-based tasks, plot their performance ranking along a axis for a specific task and obtain two sets of radar-like charts in Figure 1.
Revised: We evaluate MEC together with multiple previous SSL representations on the four image-based tasks and the five video-based tasks, plot their performance ranking along a axis for a specific task and obtain two sets of radar-like charts in Figure 1.

Original: Intuitively, it can be seen that the larger a polygon in a radar chart is, the better the corresponding representation generalize across downstream tasks.
Revised: Intuitively, it can be seen that the larger a polygon in a radar chart is, the better the corresponding representation generalize across downstream tasks.

Original: MEC consistently outperforms previous representations on both image- and video-based tasks considered, suggesting its good generalization capacity to various downstream tasks and data distributions.
Revised: MEC consistently outperforms previous representations on both image- and video-based tasks considered, suggesting its good generalization capacity to various downstream tasks and data distributions.

Original: Below we detail experiments on each task.
Revised: Below we detail experiments on each task.

Original: And additional experiment results can be found in Appendix D, including pre-training with different strategies and datasets, transfer learning on fine-grained classification benchmarks.
Revised: And additional experiment results can be found in Appendix D, including pre-training with different strategies and datasets, transfer learning on fine-grained classification benchmarks.

Original: 3.2 Evaluation on ImageNet
Revised: 3.2 Evaluation on ImageNet

Original: Linear probing.
Revised: Linear probing.

Original: We train a linear classifier on top of frozen representations of the pre-trained model on the ImageNet training set, and report the top-1 accuracy on the ImageNet validation set, which is a
Revised: We train a linear classifier on top of frozen representations of the pre-trained model on the ImageNet training set, and report the top-1 accuracy on the ImageNet validation set, which is a

Original: [Figure]
Revised: [Figure]

Original: MEC kNN linear
Revised: MEC kNN linear

Original: SimSiam [14] 83.
Revised: SimSiam [14] 83.

Original: 86.1 ✓ 84.
Revised: 86.1 ✓ 84.

Original: SimCLR [12] 84.
Revised: SimCLR [12] 84.

Original: 85.9 ✓ 87.
Revised: 85.9 ✓ 87.

Original: standard and important protocol in SSL [88, 12, 31, 14, 86].
Revised: standard and important protocol in SSL [88, 12, 31, 14, 86].

Original: We evaluate the learned representations of models with four different pre-training epochs following [14], and make comparisons in Tablewith state-of-the-art methods, whose results are reported in [14] and better than their original papers.
Revised: We evaluate the learned representations of models with four different pre-training epochs following [14], and make comparisons in Tablewith state-of-the-art methods, whose results are reported in [14] and better than their original papers.

Original: Even so, our method MEC achieves highest accuracy under all pre-training epochs among those methods.
Revised: Even so, our method MEC achieves highest accuracy under all pre-training epochs among those methods.

Original: In particular, with only 200-epoch pre-training, MEC outperforms several 800-epoch pre-trained methods, including SimCLR [12], SwAV [10] and SimSiam [14], which indicates the pretraining efficiency of our method.
Revised: In particular, with only 200-epoch pre-training, MEC outperforms several 800-epoch pre-trained methods, including SimCLR [12], SwAV [10] and SimSiam [14], which indicates the pretraining efficiency of our method.

Original: Additionally, by incorporating other pre-training strategies, such as multi-crop [10, 11], our method can attain even better results, which are detailed in Appendix D.
Revised: Additionally, by incorporating other pre-training strategies, such as multi-crop [10, 11], our method can attain even better results, which are detailed in Appendix D.

Original: Furthermore, for a fixed given downstream task ( i.e. , image classification), MEC also improves generalization across different data distributions, which can be concluded from the transfer learning experiments on 11 fine-grained classification datasets in Appendix D.
Revised: Furthermore, for a fixed given downstream task ( i.e. , image classification), MEC also improves generalization across different data distributions, which can be concluded from the transfer learning experiments on 11 fine-grained classification datasets in Appendix D.

Original: Semi-supervised classification.
Revised: Semi-supervised classification.

Original: We fine-tune the pre-trained model on a small subset of ImageNet for classification task.
Revised: We fine-tune the pre-trained model on a small subset of ImageNet for classification task.

Original: Specifically, we adopt the same fixed splits of 1% and 10% of ImageNet training set as in [12] and report both top-1 and top-5 accuracies in Table 2.
Revised: Specifically, we adopt the same fixed splits of 1% and 10% of ImageNet training set as in [12] and report both top-1 and top-5 accuracies in Table 2.

Original: The experiment results show that our method consistently outperforms previous approaches in both 1% and 10% settings.
Revised: The experiment results show that our method consistently outperforms previous approaches in both 1% and 10% settings.

Original: And we also note that all SSL methods surpass the supervised counterparts by a large margin ( e.g. , over 20% accuracy using 1% labeled data), which emphasizes the importance of leveraging large amount of unlabeled data.
Revised: And we also note that all SSL methods surpass the supervised counterparts by a large margin ( e.g. , over 20% accuracy using 1% labeled data), which emphasizes the importance of leveraging large amount of unlabeled data.

Original: 3.3 Transfer learning
Revised: 3.3 Transfer learning

Original: Object detection and instance segmentation.
Revised: Object detection and instance segmentation.

Original: We transfer the pre-trained model by end-to-end fine-tuning on object detection and instance segmentation tasks, following the common practice of previous methods [31, 13, 14, 29, 86].
Revised: We transfer the pre-trained model by end-to-end fine-tuning on object detection and instance segmentation tasks, following the common practice of previous methods [31, 13, 14, 29, 86].

Original: We report the results in Table 3.
Revised: We report the results in Table 3.

Original: Similar to the observations in semi-supervised classification, the SSL methods outperform or are on par with the supervised counterparts in all tasks.
Revised: Similar to the observations in semi-supervised classification, the SSL methods outperform or are on par with the supervised counterparts in all tasks.

Original: Our method achieves better performance than previous approaches in most of the tasks and metrics considered.
Revised: Our method achieves better performance than previous approaches in most of the tasks and metrics considered.

Original: In particular, MEC is significantly better than the supervised baseline (+1.6 AP), SimCLR (+1.9 AP), and previous best method MoCo v2 (+0.5 AP) on COCO detection task.
Revised: In particular, MEC is significantly better than the supervised baseline (+1.6 AP), SimCLR (+1.9 AP), and previous best method MoCo v2 (+0.5 AP) on COCO detection task.

Original: These results validate that the model pre-trained by MEC can generalize very well across different image-based tasks and datasets.
Revised: These results validate that the model pre-trained by MEC can generalize very well across different image-based tasks and datasets.

Original: Video-based tasks.
Revised: Video-based tasks.

Original: To further evaluate whether MEC’s representation generalizes beyond imagebased tasks, we transfer the pre-trained model to a series of video tasks based on a recently proposed evaluation platform named UniTrack [76].
Revised: To further evaluate whether MEC’s representation generalizes beyond imagebased tasks, we transfer the pre-trained model to a series of video tasks based on a recently proposed evaluation platform named UniTrack [76].

Original: In contrast to above evaluation processes, UniTrack does not require any additional training and hence provides a more direct comparison between different representations.
Revised: In contrast to above evaluation processes, UniTrack does not require any additional training and hence provides a more direct comparison between different representations.

Original: We report the results in Table 4 and compare different methods using radar plots in
Revised: We report the results in Table 4 and compare different methods using radar plots in

Original: Figure 1.
Revised: Figure 1.

Original: As noted in UniTrack, there is no significant correlation between linear probe accuracy and tracking performance.
Revised: As noted in UniTrack, there is no significant correlation between linear probe accuracy and tracking performance.

Original: However, our method achieves strong performance across linear probing and all five video tracking tasks.
Revised: However, our method achieves strong performance across linear probing and all five video tracking tasks.

Original: Specifically, MEC achieves the highest rank on four our of five tasks, and ranks among the top 2 methods on seven metrics out of all nine metrics.
Revised: Specifically, MEC achieves the highest rank on four our of five tasks, and ranks among the top 2 methods on seven metrics out of all nine metrics.

Original: These results substantiate that the learned representations of MEC are more general-purpose and versatile.
Revised: These results substantiate that the learned representations of MEC are more general-purpose and versatile.

Original: We attribute this important property of MEC to the principle of maximum entropy, the basis of our method, which enables the model to make best use of data for unseen downstream tasks during pre-training.
Revised: We attribute this important property of MEC to the principle of maximum entropy, the basis of our method, which enables the model to make best use of data for unseen downstream tasks during pre-training.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3.4 Ablation studies
Revised: 3.4 Ablation studies

Original: Batch size and training epoch.
Revised: Batch size and training epoch.

Original: The computation and memory resources required for training self-supervised models are demanding.
Revised: The computation and memory resources required for training self-supervised models are demanding.

Original: Previous methods often need a large batch size ( e.g. ,in SimCLR [12]) and long training time ( e.g. , 1000 epochs in Barlow Twins [86]) to work well.
Revised: Previous methods often need a large batch size ( e.g. ,in SimCLR [12]) and long training time ( e.g. , 1000 epochs in Barlow Twins [86]) to work well.

Original: We hence test the efficiency of our method by pre-training the model with smaller training epochs (Table 5a) and batch sizes (Table 5b).
Revised: We hence test the efficiency of our method by pre-training the model with smaller training epochs (Table 5a) and batch sizes (Table 5b).

Original: The results show that our method can effectively avoid those prohibitive requirements.
Revised: The results show that our method can effectively avoid those prohibitive requirements.

Original: In particular, Our 50-epoch pre-trained model has already outperformed those 100-epoch pre-trained models of other methods ( e.g. , SimCLR [12], BYOL [29], Barlow
Revised: In particular, Our 50-epoch pre-trained model has already outperformed those 100-epoch pre-trained models of other methods ( e.g. , SimCLR [12], BYOL [29], Barlow

Original: Twins [86]).
Revised: Twins [86]).

Original: Moreover, MEC is more robust to different batch sizes, and even a batch size ofperforms well (with 1.1% accuracy drop).
Revised: Moreover, MEC is more robust to different batch sizes, and even a batch size ofperforms well (with 1.1% accuracy drop).

Original: These results are noticeably different from those methods based on the pretext task of instance discrimination, where a large number of negative samples is required ( e.g. , over 4% accuracy drop of SimCLR with a batch size of 128).
Revised: These results are noticeably different from those methods based on the pretext task of instance discrimination, where a large number of negative samples is required ( e.g. , over 4% accuracy drop of SimCLR with a batch size of 128).

Original: This behavior also liberates MEC from those complex designs, such as memory queue [13] and online clustering [10].
Revised: This behavior also liberates MEC from those complex designs, such as memory queue [13] and online clustering [10].

Original: Siamese networks play a critical role in discriminative SSL, and different methods often rely on different architectures to prevent mode collapse.
Revised: Siamese networks play a critical role in discriminative SSL, and different methods often rely on different architectures to prevent mode collapse.

Original: We investigate the effects of different designs of Siamese networks by varying the momentum coefficient (Table 5d) and the projector network (Table 5f).
Revised: We investigate the effects of different designs of Siamese networks by varying the momentum coefficient (Table 5d) and the projector network (Table 5f).

Original: Compared to other methods [13, 14, 29, 86], MEC still works well when the Siamese networks are direct weight-sharing (m=0) and symmetric (sym.).
Revised: Compared to other methods [13, 14, 29, 86], MEC still works well when the Siamese networks are direct weight-sharing (m=0) and symmetric (sym.).

Original: In contrast, BYOL [29] requires the momentum encoder to prevent collapse (0.3% accuracy without it), and asymmetric architectures are essential for SimSiam [14] to work.
Revised: In contrast, BYOL [29] requires the momentum encoder to prevent collapse (0.3% accuracy without it), and asymmetric architectures are essential for SimSiam [14] to work.

Original: These results demonstrate that the objective of MEC naturally avoids representational collapse without relying on special design of Siamese networks.
Revised: These results demonstrate that the objective of MEC naturally avoids representational collapse without relying on special design of Siamese networks.

Original: We also observe that the performance can be further improved with a larger projector network, consistent with the conclusions in previous methods [86, 5].
Revised: We also observe that the performance can be further improved with a larger projector network, consistent with the conclusions in previous methods [86, 5].

Original: Taylor series approximation.
Revised: Taylor series approximation.

Original: The computation of log-determinant of a high-dimensional matrix can be significantly accelerated by exploiting Taylor series approximation.
Revised: The computation of log-determinant of a high-dimensional matrix can be significantly accelerated by exploiting Taylor series approximation.

Original: And we study two important hyper-parameters of the process, i.e. , the order of Taylor expansion (Table 5e) and the distortion
Revised: And we study two important hyper-parameters of the process, i.e. , the order of Taylor expansion (Table 5e) and the distortion

Original: All methods are pre-trained with two 224 × 224 crops on ImageNet training set, and we report the results of linear probing on ImageNet validation set.
Revised: All methods are pre-trained with two 224 × 224 crops on ImageNet training set, and we report the results of linear probing on ImageNet validation set.

Original: [Table]
Revised: [Table]

Original: measure ϵ d (Table 5c).
Revised: measure ϵ d (Table 5c).

Original: We find that the accuracy increases with higher order of expansion, since it produces more accurate approximation.
Revised: We find that the accuracy increases with higher order of expansion, since it produces more accurate approximation.

Original: Besides the linear probing, we find the COCO detection performance can be improved by a large margin (+1.5 AP) when the order is increased from 1 to 4.
Revised: Besides the linear probing, we find the COCO detection performance can be improved by a large margin (+1.5 AP) when the order is increased from 1 to 4.

Original: We also note that the extra two orders (from 2 to 4) benefit less than lifting from the first order to the fourth order approximation.
Revised: We also note that the extra two orders (from 2 to 4) benefit less than lifting from the first order to the fourth order approximation.

Original: It is reasonable since the relative approximation error of second-order expansion is already quite decent, lower than 0.5%
Revised: It is reasonable since the relative approximation error of second-order expansion is already quite decent, lower than 0.5%

Original: (Figure 6), almost the same as fourth-order expansion (Figure 3).
Revised: (Figure 6), almost the same as fourth-order expansion (Figure 3).

Original: As for the distortion hyper-parameter, on one hand, it should be large enough so that the Taylor expansion is convergent, and we empirically observe this condition is satisfied over a wide range of degree of distortion.
Revised: As for the distortion hyper-parameter, on one hand, it should be large enough so that the Taylor expansion is convergent, and we empirically observe this condition is satisfied over a wide range of degree of distortion.

Original: On the other hand, a good choice of ϵ d is also needed to improve the representation quality, because it compromises the uniformity and tolerance properties of representations as depicted in Figure 4 and discussed in Section 2.2.
Revised: On the other hand, a good choice of ϵ d is also needed to improve the representation quality, because it compromises the uniformity and tolerance properties of representations as depicted in Figure 4 and discussed in Section 2.2.

Original: Generalization across different backbones and objectives.
Revised: Generalization across different backbones and objectives.

Original: Given the recent progress in Vision
Revised: Given the recent progress in Vision

Original: Transformers (ViTs) [21], it is worthwhile to examine the generalization ability of our method on both
Revised: Transformers (ViTs) [21], it is worthwhile to examine the generalization ability of our method on both

Original: ViTs and ConvNets, though previous methods usually focus on only one architecture.
Revised: ViTs and ConvNets, though previous methods usually focus on only one architecture.

Original: To this end, we make a straightforward extension by replacing the ConvNet encoder with a ViT encoder, following the practice of [15].
Revised: To this end, we make a straightforward extension by replacing the ConvNet encoder with a ViT encoder, following the practice of [15].

Original: The comparisons are made in Table 5 with previous methods based on Siamese networks [12, 29, 10, 11, 31], and also recently proposed masked auto-encoding methods [30, 90] tailored for Transformers.
Revised: The comparisons are made in Table 5 with previous methods based on Siamese networks [12, 29, 10, 11, 31], and also recently proposed masked auto-encoding methods [30, 90] tailored for Transformers.

Original: And the results show that MEC performs equally well with both architectures.
Revised: And the results show that MEC performs equally well with both architectures.

Original: Furthermore, since MEC is designed for optimizing the representation quality, it should be complementary to other methods that focus on pretext tasks.
Revised: Furthermore, since MEC is designed for optimizing the representation quality, it should be complementary to other methods that focus on pretext tasks.

Original: To validate this hypothesis, we apply MEC as a regulation term on the representations of a contrastive method, SimCLR [12], and a non-contrastive method, SimSiam [14].
Revised: To validate this hypothesis, we apply MEC as a regulation term on the representations of a contrastive method, SimCLR [12], and a non-contrastive method, SimSiam [14].

Original: The results in Figure 5 demonstrate that MEC improves both linear and kNN accuracy by a large margin ( e.g. , 2.3% linear probing accuracy for SimCLR) with negligible computation overhead ( ∼ 1.05 × running time).
Revised: The results in Figure 5 demonstrate that MEC improves both linear and kNN accuracy by a large margin ( e.g. , 2.3% linear probing accuracy for SimCLR) with negligible computation overhead ( ∼ 1.05 × running time).

Original:  Related Work
Revised:  Related Work

Original: Maximum entropy principle states that the probability distribution which best represents the current state of knowledge is the one with largest entropy, in the context of precisely stated prior data.
Revised: Maximum entropy principle states that the probability distribution which best represents the current state of knowledge is the one with largest entropy, in the context of precisely stated prior data.

Original: This principle originates in Laplace’s "Principle of Insufficient Reason" and is popularized by Jaynes [38, 39].
Revised: This principle originates in Laplace’s "Principle of Insufficient Reason" and is popularized by Jaynes [38, 39].

Original: It has a wide variety of applications in different fields [40], ranging from statistical physics [58, 51], economics [2, 62], to reinforcement learning [77, 49] and supervised learning [57, 64, 22].
Revised: It has a wide variety of applications in different fields [40], ranging from statistical physics [58, 51], economics [2, 62], to reinforcement learning [77, 49] and supervised learning [57, 64, 22].

Original: To the best of our knowledge, the usage of this principle has not been explored in the context of SSL.
Revised: To the best of our knowledge, the usage of this principle has not been explored in the context of SSL.

Original: We notice a closely related work MCR 2 [85] exploits the same coding length function (Equation (1)), but the motivation behind is quite different: they do not originate from maximum entropy, but instead base their method on coding rate reduction.
Revised: We notice a closely related work MCR 2 [85] exploits the same coding length function (Equation (1)), but the motivation behind is quite different: they do not originate from maximum entropy, but instead base their method on coding rate reduction.

Original: Furthermore, MCR 2 experiments on classification and clustering tasks, and is difficult to generalize to modern SSL settings due to the expensive and unstable computation.
Revised: Furthermore, MCR 2 experiments on classification and clustering tasks, and is difficult to generalize to modern SSL settings due to the expensive and unstable computation.

Original: In contrast, our method is tailored for SSL and we reformulate the coding length function into a scalable and stable form for large-scale pre-training.
Revised: In contrast, our method is tailored for SSL and we reformulate the coding length function into a scalable and stable form for large-scale pre-training.

Original: Another closely related concept in information theory is mutual information, which has been widely used in SSL [34, 53, 68, 67, 69] based on the infomax principle [45].
Revised: Another closely related concept in information theory is mutual information, which has been widely used in SSL [34, 53, 68, 67, 69] based on the infomax principle [45].

Original: Our method can also be viewed as an application of the infomax principle that maximizes the representation information, but without using mutual information estimators, which might result in worse transfer performance [70].
Revised: Our method can also be viewed as an application of the infomax principle that maximizes the representation information, but without using mutual information estimators, which might result in worse transfer performance [70].

Original: Pretext tasks are one of the core components in SSL.
Revised: Pretext tasks are one of the core components in SSL.

Original: They refer to a type of tasks that mine a certain structure of the data, and then force learners to predict such a structure from degenerated input data.
Revised: They refer to a type of tasks that mine a certain structure of the data, and then force learners to predict such a structure from degenerated input data.

Original: Labels are mined from the data itself, therefore dispensing with the need of human annotations.
Revised: Labels are mined from the data itself, therefore dispensing with the need of human annotations.

Original: In natural language processing, the most commonly used pretext task is masked language modeling (MLM) [18, 60], where a random portion of the input sequence is masked and the model learns to predict these masked content.
Revised: In natural language processing, the most commonly used pretext task is masked language modeling (MLM) [18, 60], where a random portion of the input sequence is masked and the model learns to predict these masked content.

Original: In computer vision, however, there emerge a much wider variety of pretext tasks [87, 88, 52, 83, 20, 80, 55, 19, 30, 4, 42], since images contain much more information than languages so that there are much more intrinsic structures to be mined as free learning signals
Revised: In computer vision, however, there emerge a much wider variety of pretext tasks [87, 88, 52, 83, 20, 80, 55, 19, 30, 4, 42], since images contain much more information than languages so that there are much more intrinsic structures to be mined as free learning signals

Original: Solving pretext tasks requires the model to understand the visual concepts present in images and thus useful representations can be learned.
Revised: Solving pretext tasks requires the model to understand the visual concepts present in images and thus useful representations can be learned.

Original: In this work, however, we argue that those handcrafted pretext tasks, like handcrafted feature engineering methods, might introduce undesired bias into the representations.
Revised: In this work, however, we argue that those handcrafted pretext tasks, like handcrafted feature engineering methods, might introduce undesired bias into the representations.

Original: So we pursue a straightforward approach to optimize on representations.
Revised: So we pursue a straightforward approach to optimize on representations.

Original: And considering that the ultimate goal of self-supervised learning is a general-purpose representation, we explicitly encourage the generalization ability on downstream tasks and minimize the bias in the formulation of the pretext task, by introducing the maximum entropy principle.
Revised: And considering that the ultimate goal of self-supervised learning is a general-purpose representation, we explicitly encourage the generalization ability on downstream tasks and minimize the bias in the formulation of the pretext task, by introducing the maximum entropy principle.

Original: Siamese representation learning uses two identical neural networks to learn representations by comparing entities, in supervised manner [8, 65, 7], or recently in self-supervised manner [12, 31, 13, 14, 29, 10, 11, 86, 5, 23, 43].
Revised: Siamese representation learning uses two identical neural networks to learn representations by comparing entities, in supervised manner [8, 65, 7], or recently in self-supervised manner [12, 31, 13, 14, 29, 10, 11, 86, 5, 23, 43].

Original: In Siamese SSL, the two branches are weight-sharing [12, 14], or one is a moving average of the other [31, 29].
Revised: In Siamese SSL, the two branches are weight-sharing [12, 14], or one is a moving average of the other [31, 29].

Original: One crucial problem that needs to be carefully addressed in Siamese SSL is mode collapse, where all data is mapped to the same representation.
Revised: One crucial problem that needs to be carefully addressed in Siamese SSL is mode collapse, where all data is mapped to the same representation.

Original: Contrastive methods, such as SimCLR [12] and MoCo [31], prevent the trivial solution by minimizing the distance between augmented views of the same images while maximizing the distance between different images.
Revised: Contrastive methods, such as SimCLR [12] and MoCo [31], prevent the trivial solution by minimizing the distance between augmented views of the same images while maximizing the distance between different images.

Original: Clustering-based methods [9, 3, 10] enforces consistency between cluster assignments of the views from same images, where the cluster centers work as negative prototypes.
Revised: Clustering-based methods [9, 3, 10] enforces consistency between cluster assignments of the views from same images, where the cluster centers work as negative prototypes.

Original: Non-contrastive methods [14, 29] remove the use of negative pairs and SimSiam [14] concludes that the stop-gradient operation is critical to preventing mode collapse.
Revised: Non-contrastive methods [14, 29] remove the use of negative pairs and SimSiam [14] concludes that the stop-gradient operation is critical to preventing mode collapse.

Original: Our work also leverages the basic view consistency prior inherent in Siamese representation learning, which gives testable information for the application of maximum entropy principle.
Revised: Our work also leverages the basic view consistency prior inherent in Siamese representation learning, which gives testable information for the application of maximum entropy principle.

Original: Compared to previous work, our method of maximum entropy coding naturally avoids the low-entropy mode collapse problem and can be easily implemented without requiring instance discrimination [20, 12], memory queue [80, 31, 13] or online clustering [9, 3, 10].
Revised: Compared to previous work, our method of maximum entropy coding naturally avoids the low-entropy mode collapse problem and can be easily implemented without requiring instance discrimination [20, 12], memory queue [80, 31, 13] or online clustering [9, 3, 10].

Original: Furthermore, we find interesting equivalence between low-order approximations of MEC and existing batch-wise or feature-wise objectives, which provides a new perspective for a unified understanding of SSL methods.
Revised: Furthermore, we find interesting equivalence between low-order approximations of MEC and existing batch-wise or feature-wise objectives, which provides a new perspective for a unified understanding of SSL methods.

Original:  Conclusion
Revised:  Conclusion

Original: What makes for generalizable representations?
Revised: What makes for generalizable representations?

Original: In this work, we argue that the best generalizable representation is supposed to be the one that admits the maximum entropy.
Revised: In this work, we argue that the best generalizable representation is supposed to be the one that admits the maximum entropy.

Original: Based on this assumption, we propose MEC, a novel pretext task that aims explicitly to generalize well.
Revised: Based on this assumption, we propose MEC, a novel pretext task that aims explicitly to generalize well.

Original: MEC pursues a representation with maximum entropy in all eligible ones that satisfies the view consistency prior of Siamese representation learning.
Revised: MEC pursues a representation with maximum entropy in all eligible ones that satisfies the view consistency prior of Siamese representation learning.

Original: The minimal coding length in lossy data coding is leveraged as a computationally tractable surrogate for the entropy; and in order to facilitate large-scale pretraining, we reformulate the coding length function into a scalable form.
Revised: The minimal coding length in lossy data coding is leveraged as a computationally tractable surrogate for the entropy; and in order to facilitate large-scale pretraining, we reformulate the coding length function into a scalable form.

Original: With the reformulation, we further demonstrate that MEC bridges the batch-wise and feature-wise SSL objectives.
Revised: With the reformulation, we further demonstrate that MEC bridges the batch-wise and feature-wise SSL objectives.

Original: Extensive experiments shows MEC generalizes well across various image/video tasks, various data distributions, and different network architectures.
Revised: Extensive experiments shows MEC generalizes well across various image/video tasks, various data distributions, and different network architectures.

Original: Furthermore, MEC can be readily plugged into existing SSL methods such as SimCLR and SimSiam, and brings consistent improvements upon these strong representations.
Revised: Furthermore, MEC can be readily plugged into existing SSL methods such as SimCLR and SimSiam, and brings consistent improvements upon these strong representations.

Original: We hope our exploration will inspire the community to rethink the possibility of optimizing generalization explicitly in self-supervised representation learning.
Revised: We hope our exploration will inspire the community to rethink the possibility of optimizing generalization explicitly in self-supervised representation learning.

Original:  Funding Transparency Statement
Revised:  Funding Transparency Statement

Original: Acknowledgement: This work is supported by the state key development program in 14th Five-Year under Grant No. 2021YFF0602103, 2021YFF0602102, 2021QY1702.
Revised: Acknowledgement: This work is supported by the state key development program in 14th Five-Year under Grant No. 2021YFF0602103, 2021YFF0602102, 2021QY1702.

Original: We also thank for the research fund under Grant No. 2019GQG0001 from the Institute for Guo Qiang, Tsinghua University.A Pseudocode of MEC [Table] B CIFAR-10 Experiments
Revised: We also thank for the research fund under Grant No. 2019GQG0001 from the Institute for Guo Qiang, Tsinghua University.

Original: In this section, we detail the setting of the preliminary experiment described in Figure 4 in the main text.

Original: We train two models with different hyperparameters ϵ = 0 .

Original: 12 and ϵ = 0 .

Original: After training, we extract representations of the CIFAR-10 training set and employ T-SNE [71] to map the representation to a two-dimensional space for visualization.

Original: Besides the hyperparameter ϵ , other training configurations are kept identical and detailed below.

Original: Following the practice in [14], we do not use blur augmentation, and adopt the CIFAR variant of ResNet-18 [33] as backbone, Specifically, we remove the first max-pooling layer of ResNet-18, and set the kernel size of the first convolution layer to 3.

Original: The last classification layer is also removed and we treat the features after global average pooling as inputs to the projector, which is a two-layer MLP with BN [37] and ReLU [50] applied.

Original: The dimension of the output representation is 2048.

Original: We use SGD optimizer with weight decay =0 .

Original: 0005 , momentum =0 . 9 , and set the base learning rate to 0.03, which is linearly scaled with the batch size of 256.

Original: The learning rate is scheduled to a cosine decay rate for 600 epochs.

Original: C Implementation Details.

Original: Data augmentations.

Original: We adopt the same set of data augmentations following the common practice of previous methods [12, 29, 86, 14, 15], which is composed of geometric, color, and blurring augmentations.

Original: The geometric augmentations include random cropping, resizing to 224 × 224 , and random horizontal flipping.

Original: The color augmentations consist of a random sequence of brightness, contrast, saturation, hue adjustments, and a grayscale conversion.

Original: The blurring augmentations include Gaussian blurring and solarization.

Original: We use the same augmentation parameters as BYOL [29] and Barlow Twins [86].

Original: For each iteration, each image is augmented twice to generate two views according to the above augmentation policy.

Original: Architecture.

Original: We use a standard ResNet-50 network [33] without the final classification layer as the backbone, which yields a feature with dimension of 2048.

Original: It is followed by a projector network, which is a three-layer MLP with BN [37] and ReLU [50] applied, and each with 2048 output units.

Original: A momentum encoder is utilized to stabilize the training and further improve the performance.

Original: We turn one branch of the Siamese networks as online network and the other as target network, whose parameters are an exponential moving average of the online parameters.

Original: For the asymmetric network design, we append a two-layer MLP only to the online branch, and it has hidden dimension 512 and output dimension 2048 with BN [37] and ReLU [50] applied to the first layer.

Original: The output embeddings of the two branches are fed to the objective function for self-supervised pre-training.

Original: And after pre-training, we only keep the encoder for downstream tasks.

Original: Optimization.

Original: We use the SGD optimizer with a cosine decay learning rate schedule [46] and a linear warm-up period of 10 epochs.

Original: [-The weight decay is 1 . 
-]
Revised: [-The weight decay is 1 . 
-]

Original: × 10 − 5 and the momentum is 0 .

Original: We set the base learning rate to 0 .

Original: 5 , which is scaled linearly [28] with a batch size of 256 ( i.e. ,

Original: LearningRate = 0 .

Original: 5 × BatchSize/ 256 ).

Original: The exponential moving average parameter is increased from 0.996 to 1 with a cosine scheduler.

Original: We set the level of distortion ϵ 2 d = 0 .

Original: 06 and use a batch size of 1024.

Original: To enable large-batch and faster pre-training of 800 epochs, we adopt the LARS optimizer [84] with a batch size of 4096 and set the base learning rate to 0 .

Original: 3 and weight decay to 1 .

Original: 5 × 10 − 6 .

Original: We use the 800-epoch pre-trained model for downsteam tasks.

Original: To give an intuition of the computation overhead of our method, it takes MEC 42 hours for 100-epoch pre-training on 8 V100 GPUs, while it takes BYOL and Barlow Twins 45 and 48 hours on the same hardware.

Original: Linear probing.

Original: We adopt the standard linear probing protocol [12, 14, 29, 86] and train a supervised linear classifier on top of the frozen representation.

Original: We use the LARS optimizer [84] with a batch size of 4096 and a cosine learning rate schedule over 100 epochs.

Original: The momentum is 0.9 and the wight decay is set to 0.

Original: During training, the input images are augmented by taking a random crop, resizing to 224 × 224 , and flipping horizontally.

Original: At test time, we resize the image to 256 × 256 and then center-crop it to a size of 224 × 224 .

Original: Semi-supervised classification.

Original: We follow the semi-supervised learning protocol of [12, 86, 29] and fine-tune the pre-trained model on the 1% and 10 % subset of ImageNet [17] training set, using the same splits as in SimCLR [12].

Original: We use the SGD optimizer with a batch size of 1024 and a momentum of 0.9.

Original: And we set the weight decay to 0.

Original: We use a base learning rate of 0.05 and fine-tune the model for 50 epochs.

Original: The data augmentations are the same as in linear probing.

Original: Object detection and instance segmentation.

Original: We follow the common practice of previous methods [31, 13, 14, 86] and evaluate the transfer learning performance based on Detectron2 library [79].

Original: We initialize the backbone ResNet-50 for Faster R-CNN [61] and Mask R-CNN [32] using our pre-trained model.

Original: All Faster/Mask R-CNN models are with the C4-backbone.

Original: We fine-tune the model end-to-end in the target datasets with a searched learning rate and keep all other parameters the same as in Detectron2 library [79].

Original: We use the VOC07+12 trainval set of 16K images for training the Faster R-CNN model for 24K iterations using a batch size of 16 across 8 GPUs.

Original: The initial learning rate is reduced by a factor of 10 after 18K and 22K iterations.

Original: We also train the model using only the VOC07 trainval set of 5K images with smaller iterations according to the dataset size.

Original: We report results on VOC07 test averaged over 5 runs.

Original: We train the Mask R-CNN model (1 × schedule) on the COCO 2017 train split and report results on the val split.

Original: Object tracking.

Original: We further evaluate the generalization capability of the learned representations on five video tasks, including single object tracking (SOT) [78], video object segmentation (VOS) [56], multi-object tracking (MOT) [48], multi-object tracking and segmentation (MOTS) [73] and pose tracking (PoseTrack) [1].

Original: The datasets and metrics used for the above tasks are as follows:

Original: Task

Original: SOT

Original: VOS

Original: MOT

Original: MOTS

Original: PoseTrack

Original: Dataset

Original: OTB 2015 [78]

Original: DAVIS 2017 [56]

Original: MOT 16 [48]

Original: MOTS [73]

Original: PoseTrack 2017 [1]

Original: Metrics

Original: AUC

Original: J -mean

Original: IDF1 HOTA

Original: IDF1 HOTA

Original: IDF

Original: [Equation]

Original: -switch (IDs)

Original: All evaluations are based on the platform of UniTrack [76], where no additional fine-tuning is required.

Original: UniTrack [76] consists of a single and task-agnostic appearance model, which is initialized using our pre-trained model, and multiple heads to directly address different tasks without further training.

Original: Details of Figure 1.

Original: In Figure 1 of the main paper, we make a comparison of transfer learning performance on five image-based tasks and five video-based tasks.

Original: The image-based tasks include linear probing (top-1 accuracy) with 800-epoch pre-trained models (LIN), semi-supervised classification (top-1 accuracy) using 1% subset of training data (SEMI), object detection (AP) on VOC dataset (VOC) and COCO dataset (COCO), instance segmentation (AP mask ) on COCO dataset (SEG).

Original: For video-based tasks, we compute rankings in terms of AUC, J -mean, IDF-1, IDF-1 and IDF-1 for

Original: SOT, VOS, MOT, PoseTracking, and MOTS, respectively.

Original: D Additional Results

Original: [Figure]

Original: Approximation with low-order expansion.

Original: In Section 2.1 of the main paper, we make a comparison between original Equation (1) and our approximation using four terms of Equation (2).

Original: In this section, we provide additional comparisons using lower-order approximations.

Original: As can be seen in Figure 6, the computation process of coding length function can be even faster ( e.g. , 0.11ms v.s. 35.48ms for dim 2048) with first-order approximation.

Original: But we also notice that the relative approximation error increases to 2 .

Original: 56% , while it is 0 .

Original: 22% and 0 .

Original: 07% for second-order and fourth-order approximation, respectively.

Original: Such approximation errors may account for the reason of performance drop on linear probing (Table 5e) and other downsteam tasks (Section 3.4).

Original: [Table]

Original: Pre-training with additional strategies.

Original: In Section 3.2 of the main paper, we make a comparison of linear probing results with different methods.

Original: And in Table 1, each method is pre-trained with two 224 × 224 views for a fair comparison.

Original: We notice that there are multiple other strategies for self-supervised pre-training, so we provide additional experiment results in Table 7 by incorporating two widely used strategies, i.e. , multi-crop augmentation [10, 11] and large projector network [86, 5, 66].

Original: Multi-crop augmentation uses additional smaller crops as local views (six 96 × 96 views following [10]).

Original: A larger projector network increases the dimension of each MLP layer (from 2048 to 8192 following [86, 5, 66]).

Original: We find these strategies can steadily improve the performance of MEC at the cost of more computation overhead.

Original: Transfer learning across different image domains.

Original: In the experiments (Section 3) of the main paper, we have evaluated the transfer learning performance on a wide variety of image- and video-

Original: [Table]

Original: based downstream tasks.

Original: In this section, to further evaluate whether the learned representations can generalize across different image domains, we transfer the pre-trained model to other classification tasks by linear probing and fine-tuning on 11 datasets.

Original: The results in Table 8 demonstrate that MEC’s representation is more generalizable and less biased compared to the supervised baseline and other models pre-trained with specific pretext tasks, consistent with the observations in the main paper.

Original: [Table]

Original: Pre-training with different datasets.

Original: We perform self-supervised pre-training on Places365 [89] dataset using the proposed method, and then linear evaluation is conducted on Places365 and

Original: ImageNet dataset.

Original: We list the experiment results in Table 9.

Original: These results show that MEC can still learn good representations when pre-trained on different kinds of datasets, and also achieve better performance than previous methods.

Original: E More Detailed Proofs

Original: Proof of Equation (2) .

Original: First, we rewrite Equation (1) by substituting µ = m + dand λ = dmϵ 2 , and then we can obtain the following simplified equation,

Original: [Equation]

Original: Next, we utilize the following identical equation [35],

Original: [Equation]

Original: and then we take logarithm of the both side of the above equation, which gives,

Original: [Equation]

Original: Let A = log (cid:16) I m + λ Z ⊤ Z (cid:17) , then we have,

Original: [Equation]

Original: So Equation (5) can be reformulated as,

Original: [Equation]

Original: Finally, we apply Taylor series expansion to expand the logarithm of the matrix in the above equation, and obtain Equation (2),

Original: [Equation]

Original: with convergence condition: (cid:13)(cid:13)(cid:13) λ

Original: Z ⊤ Z

Original: [Equation]

Original: 2 < 1 .

Original: Proof of convergence condition of Equation (3) .

Original: To ensure the convergence of Equation (3), we require,

Original: [Equation]

Original: where C = λ Z ⊤ 1 Z 2 and λ = dmϵ 2 = 1 mϵ 2 d .

Original: We note the inequality between matrix norms,

Original: [Equation]

Original: which is a special case of Hölder’s inequality.

Original: Since 1-norm of matrix is simply the maximum absolute column sum of the matrix, we have

Original: [Equation]

Original: Note that the columns of Z 1 and Z 2 are ℓ

Original: 2 -normalized embeddings, so we have,

Original: [Equation]

Original: [Equation]

Original: Similarly, we can obtain,

Original: [Equation]

Original: [Equation]

Original: Finally, we go back to Equation (12) and obtain,

Original: [Equation]

Original: [Equation]

Original: To ensure that the convergence condition of Equation (11) can be strictly satisfied, we require,

Original: [Equation]

Original: [Equation]

Original: or we can equally set ϵ 2 d > 1 by adjusting the degree of distortion.

Original: Note that in the Section 2.2 of the main paper, we simply set ϵ 2 d = dm for d > m , which is the case of the preliminary experiments on CIFAR-10 [41] (the feature dimension d = 2048 and the batch size m = 1024 ).

Original: In practice, we empirically find that the Taylor expansion converges over a wide range of ϵ d (see Figure 4(c) and

Original: Table 5c).

Original: Proof of Equation (4) .

Original: Equation (4) is a direct result of Sylvester’s determinant identity, which states that if A and B are matrices of sizes m × d and d × m , then we have,

Original: [Equation]

Original: 18) and it can be proved by the following derivation,

Original: [Equation]

Original: and we also have,

Original: [Equation]

Original: So Equation (18) is proved.

Original: Let A = λ Z ⊤ 1 and B = Z 2 , and using the fact that the determinant of the transpose of a square matrix is equal to the determinant of the matrix, then we can prove

Original: Equation (4),

Original: [Equation]

Original: Relation to SimSiam [14] and BYOL [29].

Original: SimSiam [14] uses negative cosine similarity as loss function.

Original: And it is equivalent to the mean squared error of ℓ 2 -normalized vectors, up to a scale of 2, which is the loss used in BYOL [29].

Original: We write the loss function of SimSiam [14] as the following equation,

Original: [Equation]

Original: where z i 1 and z i 2 are the embeddings of two views of the same image i .

Original: By taking Taylor expansion (Equation (2)) of the left side of Equation (4), we obtain,

Original: [Equation]

Original: which is equivalent to Equation (22) up to a scale of µλ .

Original: Since µλ is a constant and can be absorbed by adjusting the learning rate during optimization, the objective function of SimSiam [14] and

Original: BYOL [29] can be viewed as the first order expansion of the objective function of MEC.

Original: Relation to Barlow Twins [86] and VICReg [5].

Original: Barlow twins [86] aims to make the crosscorrelation matrix computed from twin embeddings as close to the identity matrix as possible, with an invariance term and a redundancy reduction term.

Original: VICReg [5] follows the similar idea by using a term that decorrelates each pair of variables along each branch of Siamese networks.

Original: The objective function of Barlow twins [86] is as follows,

Original: [Equation]

Original: where C is the feature-wise cross-correlation matrix and λ barlow is a positve constant.

Original: By taking

Original: Taylor expansion (Equation (2)) of the right side of Equation (4), we obtain,

Original: [Equation]

Original: where C = λ

Original: Z 1 Z ⊤ 2 .

Original: And the above equations show that the objective function of Equation (24) can be viewed as the second-order expansion of the objective function of MEC.

Original: We notice that Barlow twins [86] uses batch normalization rather than ℓ 2 normalization on the embeddings z , and we show that these two kinds of normalization techniques have similar effects on both Barlow twins [86] and our MEC:

Original: [Table]

Original: Relation to SimCLR [12] and MoCo [13].

Original: SimCLR [12] and MoCo [13] are two typical contrastive learning methods that aim to push negative pairs apart while pulling positive pairs together.

Original: And they use the following InfoNCE loss function [53],

Original: [Equation]

Original: where τ is the temperature parameter.

Original: By taking Taylor expansion (Equation (2)) of the left side of

Original: Equation (4), we obtain,

Original: [Equation]

Original: We notice that although the above two equations do not take the exactly same forms, they have similar effects on the learning process: the first term aims to model the invariance with respect to data augmentations; and the second term aims to minimize the similarity between negative samples.

Original: F Limitations and Future Work

Original: The estimation of entropy from a finite set of high-dimensional vectors is itself a challenging problem in the field of statistical learning.

Original: So in this work, as an exploration of the principle of maximum entropy in self-supervised learning, we opt to exploit a computationally tractable surrogate for the entropy of representations.

Original: In order to facilitate large-scale pre-training, we further leverage

Original: Taylor series expansion to accelerate the computation process and we notice that more theoretical investigation is needed for the empirical convergence of small distortion.

Original: Future work can seek a more direct entropy estimator for maximum entropy coding.

Original: The proposed method aims to alleviate the bias introduced by the specific pretext task, and we notice that the bias can also be introduced by the designed data augmentations, which is a common problem in current self-supervised learning methods.

Original: Future work may seek automated data augmentation strategies and further generalize the proposed method to other modalities ( e.g. , audio, text).

Original: G Broader Impact

Original: As the first method that introduces the principle of maximum entropy into self-supervised learning, the presented work may inspire more methods leveraging this principle towards learning generalizable representations, which is the core of self-supervised learning.

Original: As we demonstrate in the experiments, the proposed method positively contributes to a wide variety of vision tasks, such as image classification, object detection and tracking.

Original: However, there is also a potential that the proposed method has negative societal impacts for a particular use of the applications.

Original: The proposed method learns representations from large-scale datatsets and the learned representations may reflect the data biases inherent in the datasets.

Original: H Licenses of Assets

Original: CIFAR-10 [41] is subject to MIT license.

Original: VOC [25] data includes images obtained from the Flickr website and use of these images is subject to the Flickr terms of use.

Original: COCO [44] is subject to the

Original: Creative Commons Attribution 4.0 License.

Original: ImageNet [17] is subject to the licenses on the website 2 .