Original: D ISENTANGLING P ROPERTIES OF C ONTRASTIVE
Revised: D ISENTANGLING P ROPERTIES OF C ONTRASTIVE

Original: M ETHODS
Revised: M ETHODS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Disentangled representation learning is an important topic in representation learning, since it not only allows the representation to be human interpretable, but it is also robust and beneﬁts downstream task performance.
Revised: Disentangled representation learning is an important topic in representation learning, since it not only allows the representation to be human interpretable, but it is also robust and beneﬁts downstream task performance.

Original: Prior methods achieved initial successes on simplistic synthetic datasets but failed to scale to complex real-world datasets.
Revised: Prior methods achieved initial successes on simplistic synthetic datasets but failed to scale to complex real-world datasets.

Original: Most of the previous methods adopt image generative models, such as GAN and VAE, to learn the disentangled representation.
Revised: Most of the previous methods adopt image generative models, such as GAN and VAE, to learn the disentangled representation.

Original: But we observe they are hard to learn disentangled representation on real-world images.
Revised: But we observe they are hard to learn disentangled representation on real-world images.

Original: Recently, self-supervised contrastive methods such as MoCo, SimCLR, and BYOL have achieved impressive performances on large-scale visual recognition tasks.
Revised: Recently, self-supervised contrastive methods such as MoCo, SimCLR, and BYOL have achieved impressive performances on large-scale visual recognition tasks.

Original: In this paper, we explored the possibility of using contrastive methods to learn a disentangled representation, a discriminative approach that is drastically different from previous approaches.
Revised: In this paper, we explored the possibility of using contrastive methods to learn a disentangled representation, a discriminative approach that is drastically different from previous approaches.

Original: Surprisingly, we ﬁnd that the contrastive method learns a disentangled representation with only minor modiﬁcations.
Revised: Surprisingly, we ﬁnd that the contrastive method learns a disentangled representation with only minor modiﬁcations.

Original: The contrastively learned representation satisﬁes a “group disentanglement” property, which is a relaxed version of the original disentanglement property.
Revised: The contrastively learned representation satisﬁes a “group disentanglement” property, which is a relaxed version of the original disentanglement property.

Original: This relaxation might be useful for scaling disentanglement learning to large and complex datasets.
Revised: This relaxation might be useful for scaling disentanglement learning to large and complex datasets.

Original: We further ﬁnd contrastive methods achieve state-of-thet-art disentanglement performance on several widely used benchmarks, such as dSprites and Car3D. It also achieves signiﬁcantly higher performance on the real-world dataset CelebA.
Revised: We further ﬁnd contrastive methods achieve state-of-thet-art disentanglement performance on several widely used benchmarks, such as dSprites and Car3D. It also achieves signiﬁcantly higher performance on the real-world dataset CelebA.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning a disentangled representation is a long-desired goal in the deep learning community (Bengio et al., 2013; Peters et al., 2017; Goodfellow et al., 2016; Bengio et al., 2007; Schmidhuber, 1992;
Revised: Learning a disentangled representation is a long-desired goal in the deep learning community (Bengio et al., 2013; Peters et al., 2017; Goodfellow et al., 2016; Bengio et al., 2007; Schmidhuber, 1992;

Original: Lake et al., 2017; Tschannen et al., 2018).
Revised: Lake et al., 2017; Tschannen et al., 2018).

Original: A disentangled representation matches how humans understand the world and provides many other beneﬁts besides model interpretability (Bengio et al., 2013; Chen et al., 2016; Kulkarni et al., 2015).
Revised: A disentangled representation matches how humans understand the world and provides many other beneﬁts besides model interpretability (Bengio et al., 2013; Chen et al., 2016; Kulkarni et al., 2015).

Original: And it usually needs much fewer labels to learn challenging downstream tasks (van Steenkiste et al., 2019).
Revised: And it usually needs much fewer labels to learn challenging downstream tasks (van Steenkiste et al., 2019).

Original: It also generalizes much better even in face of examples generated by an unseen combination of the attribute values (Achille et al., 2018).
Revised: It also generalizes much better even in face of examples generated by an unseen combination of the attribute values (Achille et al., 2018).

Original: Given its importance and potential large impacts on downstream applications, disentangled representation learning has recently attracted great attention.
Revised: Given its importance and potential large impacts on downstream applications, disentangled representation learning has recently attracted great attention.

Original: Previous research has proposed a lot of methods, either built on variational auto-encoders (Kingma & Welling, 2013), such as β -VAE (Higgins et al., 2016) and FactorVAE (Kim & Mnih, 2018), or generative adversarial network (Goodfellow et al., 2014), such as InfoGAN (Chen et al., 2016) and InfoGAN-CR (Lin et al., 2020).
Revised: Previous research has proposed a lot of methods, either built on variational auto-encoders (Kingma & Welling, 2013), such as β -VAE (Higgins et al., 2016) and FactorVAE (Kim & Mnih, 2018), or generative adversarial network (Goodfellow et al., 2014), such as InfoGAN (Chen et al., 2016) and InfoGAN-CR (Lin et al., 2020).

Original: Those methods have achieved preliminary successes on synthetic datasets such as dSprites (Matthey et al., 2017) and 3Dshapes (Burgess & Kim, 2018).
Revised: Those methods have achieved preliminary successes on synthetic datasets such as dSprites (Matthey et al., 2017) and 3Dshapes (Burgess & Kim, 2018).

Original: Albeit the initial successes, those synthetic datasets are limited in many aspects, for example, the background is usually clean and composed of a single color, the number of objects is small and the objects are mainly 2D without texture and occlusions.
Revised: Albeit the initial successes, those synthetic datasets are limited in many aspects, for example, the background is usually clean and composed of a single color, the number of objects is small and the objects are mainly 2D without texture and occlusions.

Original: It is still an open question to scale those models to real-world complex datasets.
Revised: It is still an open question to scale those models to real-world complex datasets.

Original: It is even non-trivial to scale basic generative models to complex datasets to learn disentangled features, such as ImageNet (Deng et al., 2009).
Revised: It is even non-trivial to scale basic generative models to complex datasets to learn disentangled features, such as ImageNet (Deng et al., 2009).

Original: In this paper, instead of studying disentanglement feature learning with a generative model, we investigate whether the discriminatively-trained contrastive models have the disentanglement property.
Revised: In this paper, instead of studying disentanglement feature learning with a generative model, we investigate whether the discriminatively-trained contrastive models have the disentanglement property.

Original: Contrastive learning is a class of self-supervised learning methods that pull two augmentations of the same image close.
Revised: Contrastive learning is a class of self-supervised learning methods that pull two augmentations of the same image close.

Original: The recent contrastive methods have achieved state-of-the-art performance on the image pretraining tasks.
Revised: The recent contrastive methods have achieved state-of-the-art performance on the image pretraining tasks.

Original: Representative methods include MoCo (He et al., 2020), BYOL (Grill et al., 2020), SimCLR (Chen et al., 2020) and SwAV (Caron et al., 2020) etc.
Revised: Representative methods include MoCo (He et al., 2020), BYOL (Grill et al., 2020), SimCLR (Chen et al., 2020) and SwAV (Caron et al., 2020) etc.

Original: Contrastive learning has been proven to learn good visual representations from large-scale datasets.
Revised: Contrastive learning has been proven to learn good visual representations from large-scale datasets.

Original: We continue to investigate the disentanglement property of their learned representations in this work.
Revised: We continue to investigate the disentanglement property of their learned representations in this work.

Original: To our surprise, we ﬁnd that the widely used BYOL algorithm without any auxiliary loss exhibits strong feature disentanglement property.
Revised: To our surprise, we ﬁnd that the widely used BYOL algorithm without any auxiliary loss exhibits strong feature disentanglement property.

Original: However, the disentanglement of contrastive learning is a weaker form of disentanglement.
Revised: However, the disentanglement of contrastive learning is a weaker form of disentanglement.

Original: It follows the pattern that a representation dimension is disentangled to correspond to a single factor but a single ground truth factor might appear in multiple latent feature dimensions.
Revised: It follows the pattern that a representation dimension is disentangled to correspond to a single factor but a single ground truth factor might appear in multiple latent feature dimensions.

Original: We name this type of disentanglement as “group disentanglement”.
Revised: We name this type of disentanglement as “group disentanglement”.

Original: Although group disentanglement is a weaker form of disentanglement, we hypothesize that directly learning a compact and disentangled representation might be hard, due to the lottery ticket hypothesis (Frankle & Carbin, 2018).
Revised: Although group disentanglement is a weaker form of disentanglement, we hypothesize that directly learning a compact and disentangled representation might be hard, due to the lottery ticket hypothesis (Frankle & Carbin, 2018).

Original: Pursuing group disentanglement instead of full disentanglement might be necessary to achieve disentanglement on the complex real-world dataset.
Revised: Pursuing group disentanglement instead of full disentanglement might be necessary to achieve disentanglement on the complex real-world dataset.

Original: The reason is probably that real-world images usually contain more details and noise, and a “factor” might be always correlated to some other visual existence due to dataset bias.
Revised: The reason is probably that real-world images usually contain more details and noise, and a “factor” might be always correlated to some other visual existence due to dataset bias.

Original: A good example is a ship that usually comes together with a large area of blue background in existing datasets.
Revised: A good example is a ship that usually comes together with a large area of blue background in existing datasets.

Original: Further, we ﬁnd that contrastively trained representation achieves the state-of-the-art FactorVAE disentanglement score when evaluated on established benchmarks, such as Car3D (Reed et al., 2015), dSprites (Matthey et al., 2017) and SmallNORB (LeCun et al., 2004).
Revised: Further, we ﬁnd that contrastively trained representation achieves the state-of-the-art FactorVAE disentanglement score when evaluated on established benchmarks, such as Car3D (Reed et al., 2015), dSprites (Matthey et al., 2017) and SmallNORB (LeCun et al., 2004).

Original: Besides the widely used synthetic benchmarks, we also evaluate the contrastive method on the CelebA (Liu et al., 2018) human faces real-world dataset.
Revised: Besides the widely used synthetic benchmarks, we also evaluate the contrastive method on the CelebA (Liu et al., 2018) human faces real-world dataset.

Original: We ﬁnd that it also achieves better or comparable performances than the other methods on ﬁve commonly used disentanglement metrics.
Revised: We ﬁnd that it also achieves better or comparable performances than the other methods on ﬁve commonly used disentanglement metrics.

Original: In summary, our contributions in this paper are:
Revised: In summary, our contributions in this paper is mainly empirical and include the follows:

Original: We show that a contrastive method, in particular BYOL, learns “group-disentangled” representations, without any extra auxiliary losses.
Revised: We show that a contrastive method, in particular BYOL, learns “group-disentangled” representations, without any extra auxiliary losses.

Original: The contrastive method achieves the state-of-the-art performance on several widely used disentanglement learning benchmarks.
Revised: The contrastive method achieves the state-of-the-art performance on several widely used disentanglement learning benchmarks.

Original: We propose to quantitatively evaluate disentanglements on a real-world dataset, which avoids the biases of synthetic images.
Revised: We propose to quantitatively evaluate disentanglements on a real-world dataset, which avoids the biases of synthetic images.

Original: Our contrastive method also achieves state-of-the-art performance on this benchmark.
Revised: Our contrastive method also achieves state-of-the-art performance on this benchmark.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Disentangled Representation Learning Disentangled representation is desired as it represents a human interpretable pattern (Bengio et al., 2013; Chen et al., 2016; Kulkarni et al., 2015), enabling the downstream tasks learned more easily (van Steenkiste et al., 2019) and generalizes better (Achille et al., 2018).
Revised: Disentangled Representation Learning Disentangled representation is desired as it represents a human interpretable pattern (Bengio et al., 2013; Chen et al., 2016; Kulkarni et al., 2015), enabling the downstream tasks learned more easily (van Steenkiste et al., 2019) and generalizes better (Achille et al., 2018).

Original: In this paper, we consider the fully unsupervised disentangled representation learning setting, i.e. we assume no annotations on which factors should be learned.
Revised: In this paper, we consider the fully unsupervised disentangled representation learning setting, i.e. we assume no annotations on which factors should be learned.

Original: We notice the recent study of disentanglement is promoted by two communities: Disentanglement in Deep Features and Independent Component Analysis.
Revised: We notice the recent study of disentanglement is promoted by two communities: Disentanglement in Deep Features and Independent Component Analysis.

Original: Their research previously lie on different assumptions, data patterns, and evaluation metrics.
Revised: Their research previously lie on different assumptions, data patterns, and evaluation metrics.

Original: One community is motivated by the newly raised deep learning for encouraging disentangled representation over independent factors.
Revised: One community is motivated by the newly raised deep learning for encouraging disentangled representation over independent factors.

Original: they have shown much empirical progress on this problem and they directly term their goal as “disentanglement”.
Revised: they have shown much empirical progress on this problem and they directly term their goal as “disentanglement”.

Original: The related study is usually based on deep generative models.
Revised: The related study is usually based on deep generative models.

Original: For instance, VAE-based methods have achieved successes on this task (Higgins et al., 2016;
Revised: For instance, VAE-based methods have achieved successes on this task (Higgins et al., 2016;

Original: Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2017).
Revised: Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2017).

Original: Besides, Generative Adversarial
Revised: Besides, Generative Adversarial

Original: Networks (GAN) (Goodfellow et al., 2014; Chen et al., 2016) are also put into the discussion of encouraging representations’ disentanglement.
Revised: Networks (GAN) (Goodfellow et al., 2014; Chen et al., 2016) are also put into the discussion of encouraging representations’ disentanglement.

Original: More recently, people have shown that the GAN-based approach can achieve competitive performance as the above VAE variants (Lin et al., 2020; Jeon et al., 2018; Lee et al., 2020b).
Revised: More recently, people have shown that the GAN-based approach can achieve competitive performance as the above VAE variants (Lin et al., 2020; Jeon et al., 2018; Lee et al., 2020b).

Original: A recent work (Locatello et al., 2019) summarizes the popular methods and metrics in this community and proposes a tool for evaluation called disentanglement lib , including popular metrics such as DCI (Eastwood & Williams, 2018), SAP (Kumar et al., 2017),
Revised: A recent work (Locatello et al., 2019) summarizes the popular methods and metrics in this community and proposes a tool for evaluation called disentanglement lib , including popular metrics such as DCI (Eastwood & Williams, 2018), SAP (Kumar et al., 2017),

Original: MIG (Chen et al., 2018) and so on.
Revised: MIG (Chen et al., 2018) and so on.

Original: Besides this series of studies, exploring underlying factors of variation in data pattern is a longstanding goal of the Independent Component Analysis (ICA) community (Hyv¨arinen & Oja, 2000).
Revised: Besides this series of studies, exploring underlying factors of variation in data pattern is a longstanding goal of the Independent Component Analysis (ICA) community (Hyv¨arinen & Oja, 2000).

Original: They share many similarities, for example, generative models, e.g., VAEs, are recently popular in both (Khemakhem et al., 2020a; Klindt et al., 2020).
Revised: They share many similarities, for example, generative models, e.g., VAEs, are recently popular in both (Khemakhem et al., 2020a; Klindt et al., 2020).

Original: ICA usually has different assumptions with the “purely unsupervised learning” (H¨alv¨a et al., 2021).
Revised: ICA usually has different assumptions with the “purely unsupervised learning” (H¨alv¨a et al., 2021).

Original: For example, the pattern of noise (Hyvarinen &
Revised: For example, the pattern of noise (Hyvarinen &

Original: Morioka, 2016; Khemakhem et al., 2020a) or some additional auxiliary variables (Hyvarinen et al., 2019; Khemakhem et al., 2020b) can be observed.
Revised: Morioka, 2016; Khemakhem et al., 2020a) or some additional auxiliary variables (Hyvarinen et al., 2019; Khemakhem et al., 2020b) can be observed.

Original: Traditionally, ICA uses identiﬁability to assess their desired representation pattern and the popular metric is Mean Correlation Coefﬁcient (MCC).
Revised: Traditionally, ICA uses identiﬁability to assess their desired representation pattern and the popular metric is Mean Correlation Coefﬁcient (MCC).

Original: SlowVAE (Klindt et al., 2020) recently makes a great effort to connect the two branches of study but it still requires additional information such as temporal transition pattern.
Revised: SlowVAE (Klindt et al., 2020) recently makes a great effort to connect the two branches of study but it still requires additional information such as temporal transition pattern.

Original: As our study is for purely unsupervised learning and some assumptions of ICA methods can not be well matched, we mainly follow the settings and benchmark by
Revised: As our study is for purely unsupervised learning and some assumptions of ICA methods can not be well matched, we mainly follow the settings and benchmark by

Original: disentanglement lib k
Revised: disentanglement lib k

Original: (Locatello et al., 2019).
Revised: (Locatello et al., 2019).

Original: Contrastive Learning Contrastive learning methods such as SimCLR (Chen et al., 2020),
Revised: Contrastive Learning Contrastive learning methods such as SimCLR (Chen et al., 2020),

Original: MoCo (He et al., 2020) and BYOL (Grill et al., 2020) have achieved great successes to learn good visual representation from no label.
Revised: MoCo (He et al., 2020) and BYOL (Grill et al., 2020) have achieved great successes to learn good visual representation from no label.

Original: They create “views” by applying augmentations over images.
Revised: They create “views” by applying augmentations over images.

Original: They treat two views of the same image as “positive pairs”, and views of all the other images as negatives.
Revised: They treat two views of the same image as “positive pairs”, and views of all the other images as negatives.

Original: This setup is also known as examplar classiﬁcation (Dosovitskiy et al., 2014), or instance discrimination (Wu et al., 2018).
Revised: This setup is also known as examplar classiﬁcation (Dosovitskiy et al., 2014), or instance discrimination (Wu et al., 2018).

Original: Representation learned by contrastive learning has shown great transfer capabilities to downstream tasks, such as object detection and semantic segmentation.
Revised: Representation learned by contrastive learning has shown great transfer capabilities to downstream tasks, such as object detection and semantic segmentation.

Original: More recently, there are a lot of works trying to understand contrastive learning either theoretically (Wang & Isola, 2020; Arora et al., 2019; Tsai et al., 2020; Tosh et al., 2021; Tian et al., 2020b;
Revised: More recently, there are a lot of works trying to understand contrastive learning either theoretically (Wang & Isola, 2020; Arora et al., 2019; Tsai et al., 2020; Tosh et al., 2021; Tian et al., 2020b;

Original: Lee et al., 2020a) or empirically (Tian et al., 2020a; Zhao et al., 2020; Purushwalkam & Gupta, 2020).
Revised: Lee et al., 2020a) or empirically (Tian et al., 2020a; Zhao et al., 2020; Purushwalkam & Gupta, 2020).

Original: Zimmermann et al.
Revised: Zimmermann et al.

Original: (2021) suggests that the contrastive method can invert the data generation process.
Revised: (2021) suggests that the contrastive method can invert the data generation process.

Original: The conclusion is based on the analysis of Wang & Isola (2020) where negative samples are necessary and expected to be inﬁnite.
Revised: The conclusion is based on the analysis of Wang & Isola (2020) where negative samples are necessary and expected to be inﬁnite.

Original: Zimmermann et al.
Revised: Zimmermann et al.

Original: (2021) make a good bridge between contrastive learning and independent analysis and study the model identiﬁability quantitatively in terms of MCC score.
Revised: (2021) make a good bridge between contrastive learning and independent analysis and study the model identiﬁability quantitatively in terms of MCC score.

Original: Compared with that, we focus more on more direct disentanglement analysis.
Revised: Compared with that, we focus more on more direct disentanglement analysis.

Original: Our contribution is more empirical but suggests the good disentanglement property of contrastive learning even without negative samples.
Revised: Our contribution is more empirical but suggests the good disentanglement property of contrastive learning even without negative samples.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this paper, we explore whether contrastive methods learn a disentangled feature representation.
Revised: In this paper, we explore whether contrastive methods learn a disentangled feature representation.

Original: If yes, under what condition it learns a disentangled representation.
Revised: If yes, under what condition it learns a disentangled representation.

Original: There are quite a few contrastive learning algorithms proposed (Grill et al., 2020;
Revised: There are quite a few contrastive learning algorithms proposed (Grill et al., 2020;

Original: He et al., 2020; Chen et al., 2020; Caron et al., 2020).
Revised: He et al., 2020; Chen et al., 2020; Caron et al., 2020).

Original: Although they differ on some speciﬁc aspects, they all aim to pull two augmentations of one image close.
Revised: Although they differ on some speciﬁc aspects, they all aim to pull two augmentations of one image close.

Original: Without the uniformity property provided by negative samples (Wang & Isola, 2020), we ﬁnd BYOL (Grill et al., 2020) still achieves unexpected good disentangled feature representation.
Revised: Without the uniformity property provided by negative samples (Wang & Isola, 2020), we ﬁnd BYOL (Grill et al., 2020) still achieves unexpected good disentangled feature representation.

Original: 3. BYOL M ETHOD
Revised: 3. BYOL M ETHOD

Original: BYOL is an unsupervised learning method that pulls two augmentation views of the same image close (Grill et al., 2020) to learn a high-level image representation.
Revised: BYOL is an unsupervised learning method that pulls two augmentation views of the same image close (Grill et al., 2020) to learn a high-level image representation.

Original: A signiﬁcant difference of it against other contrastive learning is the absence of negative pairs during training.
Revised: A signiﬁcant difference of it against other contrastive learning is the absence of negative pairs during training.

Original: As shown in
Revised: As shown in

Original: Figure 1, for each image x , we obtain two views of it: x 1 and x 2 by data augmentations.
Revised: Figure 1, for each image x , we obtain two views of it: x 1 and x 2 by data augmentations.

Original: One of them goes through the online network stream, and the other goes through the target network stream.
Revised: One of them goes through the online network stream, and the other goes through the target network stream.

Original: The target network’s parameter is not trained by the gradient descent algorithm but set as the exponential moving average (ema) of the online network.
Revised: The target network’s parameter is not trained by the gradient descent algorithm but set as the exponential moving average (ema) of the online network.

Original: Both the online and the target stream have a representation network (encoder) and a projection network.
Revised: Both the online and the target stream have a representation network (encoder) and a projection network.

Original: The online network has an extra prediction network after the projection network.
Revised: The online network has an extra prediction network after the projection network.

Original: The online stream’s output z 1 and the target stream’s output z 2 are pulled close to each other by requiring the two vectors to have similar directions in the latent space.
Revised: The online stream’s output z 1 and the target stream’s output z 2 are pulled close to each other by requiring the two vectors to have similar directions in the latent space.

Original: More speciﬁcally, the loss function is
Revised: More speciﬁcally, the loss function is

Original: [Equation]
Revised: [Equation]

Original: In practice, the online representation network (without the projection or the prediction network) is usually the representation model for the downstream tasks.
Revised: In practice, the online representation network (without the projection or the prediction network) is usually the representation model for the downstream tasks.

Original: In this work, we follow the same convention, i.e. we study the disentanglement property of the output of the representation network.
Revised: In this work, we follow the same convention, i.e. we study the disentanglement property of the output of the representation network.

Original: [Figure]
Revised: [Figure]

Original: 3. I MPLEMENTATION D ETAILS
Revised: 3. I MPLEMENTATION D ETAILS

Original: We follow most of the implementation details in the original BYOL paper, but we ﬁnd that several details can be changed to achieve better feature disentanglement.
Revised: We follow most of the implementation details in the original BYOL paper, but we ﬁnd that several details can be changed to achieve better feature disentanglement.

Original: For full implementation details please refer to the appendix A.1.
Revised: For full implementation details please refer to the appendix A.1.

Original: Architecture To make a fair comparison, we follow the encoder architecture used in FactorVAE (Kim & Mnih, 2018) for all synthetic datasets, except that the latent code dimension is set to be 1000, since the contrastive learning requires a large latent code dimension to work (Grill et al., 2020).
Revised: Architecture To make a fair comparison, we follow the encoder architecture used in FactorVAE (Kim & Mnih, 2018) for all synthetic datasets, except that the latent code dimension is set to be 1000, since the contrastive learning requires a large latent code dimension to work (Grill et al., 2020).

Original: On CelebA, encoders are uniﬁed to be ResNet-50 (He et al., 2016).
Revised: On CelebA, encoders are uniﬁed to be ResNet-50 (He et al., 2016).

Original: Normalization By default, BYOL uses batch normalization throughout the network.
Revised: Normalization By default, BYOL uses batch normalization throughout the network.

Original: However, we ﬁnd that batch norm in encoder can reduce the feature disentanglement in some cases (see Section 5.5 for ablations).
Revised: However, we ﬁnd that batch norm in encoder can reduce the feature disentanglement in some cases (see Section 5.5 for ablations).

Original: Thus we replace all batch norm with group normalization (Wu & He, 2018).
Revised: Thus we replace all batch norm with group normalization (Wu & He, 2018).

Original: Unless otherwise stated, the group number is set to 4 in our implementation by default.
Revised: Unless otherwise stated, the group number is set to 4 in our implementation by default.

Original: Augmentation Augmentation is a crucial component for unsupervised learning methods as it provides a feasible way to create positive pairs in contrastive learning.
Revised: Augmentation Augmentation is a crucial component for unsupervised learning methods as it provides a feasible way to create positive pairs in contrastive learning.

Original: In this work, we follow the default data augmentation used in BYOL, i.e., the composition of color jitter, graying, horizontal ﬂip, gaussian blur, and random resize crop.
Revised: In this work, we follow the default data augmentation used in BYOL, i.e., the composition of color jitter, graying, horizontal ﬂip, gaussian blur, and random resize crop.

Original: We recognize that the parameter of random resize crop size is critical to learning disentangled features, which we will discuss in the ablation study.
Revised: We recognize that the parameter of random resize crop size is critical to learning disentangled features, which we will discuss in the ablation study.

Original:  M AJOR R ESULTS
Revised:  M AJOR R ESULTS

Original: In this section, we summarize our major empirical ﬁndings.
Revised: In this section, we summarize our major empirical ﬁndings.

Original: Section 5 will then continue to present experimental evidence to support these claims.
Revised: Section 5 will then continue to present experimental evidence to support these claims.

Original: 4. T HE G ROUP D ISENTANGLEMENT P ROPERTY
Revised: 4. T HE G ROUP D ISENTANGLEMENT P ROPERTY

Original: We ﬁnd that the contrastive feature representation exhibits some form of disentanglement property: different ground truth factors map to different sets of latent dimensions, but multiple latent dimensions are corresponding to a single ground truth factor.
Revised: We ﬁnd that the contrastive feature representation exhibits some form of disentanglement property: different ground truth factors map to different sets of latent dimensions, but multiple latent dimensions are corresponding to a single ground truth factor.

Original: We name this type of disentanglement as “group disentanglement”.
Revised: We name this type of disentanglement as “group disentanglement”.

Original: See Section 5.2 for experimental evidence to support this characteristic.
Revised: See Section 5.2 for experimental evidence to support this characteristic.

Original: [Figure]
Revised: [Figure]

Original: dimensions
Revised: dimensions

Original: While there is not a single widely accepted deﬁnition of the original disentanglement concept, we describe it intuitively as (1) Disentanglement : one dimension in the representation vector only represents one ground truth factor (2) Compactness : one ground truth factor only corresponds to one dimension in the representation (3) Completeness : no ground truth factor is left out in the representation.
Revised: While there is not a single widely accepted deﬁnition of the original disentanglement concept, we describe it intuitively as (1) Disentanglement : one dimension in the representation vector only represents one ground truth factor (2) Compactness : one ground truth factor only corresponds to one dimension in the representation (3) Completeness : no ground truth factor is left out in the representation.

Original: Group disentanglement relaxes the compactness requirement, but only requires the disentanglement and the completeness.
Revised: Group disentanglement relaxes the compactness requirement, but only requires the disentanglement and the completeness.

Original: Figure 2 shows how group disentanglement looks like.
Revised: Figure 2 shows how group disentanglement looks like.

Original: Group disentanglement is a weaker form of disentanglement, but we hypothesis that it might be a better proxy of disentanglement to scale to complex and realistic datasets.
Revised: Group disentanglement is a weaker form of disentanglement, but we hypothesis that it might be a better proxy of disentanglement to scale to complex and realistic datasets.

Original: Directly learning the ideal compact disentanglement representation with neural network might be very hard because of the lottery ticket hypothesis (Frankle & Carbin, 2018).
Revised: Directly learning the ideal compact disentanglement representation with neural network might be very hard because of the lottery ticket hypothesis (Frankle & Carbin, 2018).

Original: The lottery ticket hypothesis states that usually a well-trained neural network can be pruned to a much smaller network while keeping the same accuracy.
Revised: The lottery ticket hypothesis states that usually a well-trained neural network can be pruned to a much smaller network while keeping the same accuracy.

Original: However, it is much harder to directly train the same small network from scratch when randomly initialized.
Revised: However, it is much harder to directly train the same small network from scratch when randomly initialized.

Original: The hypothesis encourages a large randomly initialized network from start to contain a subset of the network that is the “winning ticket”.
Revised: The hypothesis encourages a large randomly initialized network from start to contain a subset of the network that is the “winning ticket”.

Original: We suspect that a compact disentangled representation is also hard to obtain for the same reason.
Revised: We suspect that a compact disentangled representation is also hard to obtain for the same reason.

Original: Instead, group disentanglement might be a much more realistic goal to achieve.
Revised: Instead, group disentanglement might be a much more realistic goal to achieve.

Original: 4. C OMPETITIVE P ERFORMANCE A CROSS B ENCHMARKS AND M ETRICS
Revised: 4. C OMPETITIVE P ERFORMANCE A CROSS B ENCHMARKS AND M ETRICS

Original: To quantitatively evaluate the disentanglement properties of the contrastive methods, we evaluate with existing benchmarks (Cars3D (Reed et al., 2015), dSprites (Matthey et al., 2017), SmallNORB (LeCun et al., 2004), Shapes3D (Burgess & Kim, 2018)) with various metrics.
Revised: To quantitatively evaluate the disentanglement properties of the contrastive methods, we evaluate with existing benchmarks (Cars3D (Reed et al., 2015), dSprites (Matthey et al., 2017), SmallNORB (LeCun et al., 2004), Shapes3D (Burgess & Kim, 2018)) with various metrics.

Original: We ﬁnd that although contrastive learning is not designed to learn disentangled feature representation, it still achieves better or comparable disentangled features when compared to existing specialized methods.
Revised: We ﬁnd that although contrastive learning is not designed to learn disentangled feature representation, it still achieves better or comparable disentangled features when compared to existing specialized methods.

Original: We even build benchmarks on the real-world CelebA dataset for the ﬁrst time, and the performance of contrastive learning is still robust to be top.
Revised: We even build benchmarks on the real-world CelebA dataset for the ﬁrst time, and the performance of contrastive learning is still robust to be top.

Original: See Section 5.3 for experimental support.
Revised: See Section 5.3 for experimental support.

Original: 4. B ATCH N ORM D ISCOURAGES F EATURE D ISENTANGLEMENT
Revised: 4. B ATCH N ORM D ISCOURAGES F EATURE D ISENTANGLEMENT

Original: We ﬁnd that batch norm in encoder consistently decreases the feature disentanglement level compared to no normalization layer (Section 5.5).
Revised: We ﬁnd that batch norm in encoder consistently decreases the feature disentanglement level compared to no normalization layer (Section 5.5).

Original: By keeping the batch norm in the projector and the predictor, removing batch norm in the encoder will not cause model collapsing, which agrees with the observation in previous works (Richemond et al., 2020).
Revised: By keeping the batch norm in the projector and the predictor, removing batch norm in the encoder will not cause model collapsing, which agrees with the observation in previous works (Richemond et al., 2020).

Original: On the contrary, replacing batch norm in encoder with group norm or layer norm will increase the feature disentanglement while achieving similar accuracy in downstream factor prediction.
Revised: On the contrary, replacing batch norm in encoder with group norm or layer norm will increase the feature disentanglement while achieving similar accuracy in downstream factor prediction.

Original: We notice that a similar phenomenon has been discovered before in supervised feature disentanglement.
Revised: We notice that a similar phenomenon has been discovered before in supervised feature disentanglement.

Original: For example, Bau et al.
Revised: For example, Bau et al.

Original: (2017) discovered that a network trained with batch normalization layers has less interpretable (disentangled) neurons.
Revised: (2017) discovered that a network trained with batch normalization layers has less interpretable (disentangled) neurons.

Original: We still do not fully understand this behavior, but we hypothesize that it may be caused by the shared batch statistics that make it hard for a feature to be aligned to the ground truth factor.
Revised: We still do not fully understand this behavior, but we hypothesize that it may be caused by the shared batch statistics that make it hard for a feature to be aligned to the ground truth factor.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we show the quantitative results to support the described observations above.
Revised: In this section, we show the quantitative results to support the described observations above.

Original: We ﬁrst introduce the experiment setup.
Revised: We ﬁrst introduce the experiment setup.

Original: Then we provide qualitative analysis about the learned representation pattern by contrastive learning.
Revised: Then we provide qualitative analysis about the learned representation pattern by contrastive learning.

Original: At last, we list a series of quantitative benchmark experiments to prove the good disentanglement property of BYOL on both synthetic and real-world datasets.
Revised: At last, we list a series of quantitative benchmark experiments to prove the good disentanglement property of BYOL on both synthetic and real-world datasets.

Original: Besides the main experiments in this section, more ablation experiments are provided in the appendix.
Revised: Besides the main experiments in this section, more ablation experiments are provided in the appendix.

Original: In this section, we aim to empirically understand the disentanglement properties of the contrastive method following the four questions.
Revised: In this section, we aim to empirically understand the disentanglement properties of the contrastive method following the four questions.

Original: (1) What does the contrastive method learn in its latent representation?
Revised: (1) What does the contrastive method learn in its latent representation?

Original: Is it disentangled?
Revised: Is it disentangled?

Original: Can we visualize the latent space?
Revised: Can we visualize the latent space?

Original: (2) Quantitatively, how disentangled is the representation of contrastive methods?
Revised: (2) Quantitatively, how disentangled is the representation of contrastive methods?

Original: (3) How about real-world datasets?
Revised: (3) How about real-world datasets?

Original: (4) What set of hyper-parameters best promotes the learned representation to be disentangled?
Revised: (4) What set of hyper-parameters best promotes the learned representation to be disentangled?

Original: 5. E XPERIMENTS S ETUP
Revised: 5. E XPERIMENTS S ETUP

Original: In this section, we describe the experimental setup, including the datasets we evaluate, the metrics for a quantitative study, and the previous disentangled feature learning algorithms for comparison.
Revised: In this section, we describe the experimental setup, including the datasets we evaluate, the metrics for a quantitative study, and the previous disentangled feature learning algorithms for comparison.

Original: Datasets Previous works evaluate representation disentanglement only on synthetic datasets, such as dSprites, Cars3D, SmallNORB, and Shape3D. Besides those datasets, we also include a realworld dataset CelebA. CelebA contains human celebrity faces images with 40 binary attributes annotations.
Revised: Datasets Previous works evaluate representation disentanglement only on synthetic datasets, such as dSprites, Cars3D, SmallNORB, and Shape3D. Besides those datasets, we also include a realworld dataset CelebA. CelebA contains human celebrity faces images with 40 binary attributes annotations.

Original: The attributes include ﬁne-grained properties of the human face such as whether wearing glasses or whether has wavy hair.
Revised: The attributes include ﬁne-grained properties of the human face such as whether wearing glasses or whether has wavy hair.

Original: Table 6 in Appendix explains the dataset details.
Revised: Table 6 in Appendix explains the dataset details.

Original: [Figure]
Revised: [Figure]

Original: S h a p e
Revised: S h a p e

Original: S c a l e
Revised: S c a l e

Original: P o s i t i o n _ X
Revised: P o s i t i o n _ X

Original: P o s i t i o n _ Y
Revised: P o s i t i o n _ Y

Original: F a c t o r
Revised: F a c t o r

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:       
Revised:       

Original: Metrics We follow disentanglement lib
Revised: Metrics We follow disentanglement lib

Original: (Locatello et al., 2019) to use ﬁve popular metrics to evaluate each method.
Revised: (Locatello et al., 2019) to use ﬁve popular metrics for evaluation, i e., BetaVAE, FactorVAE, MIG, SAP, and DCI.

Original: We leave out the Modularity metric since Locatello et al.
Revised: We leave out the Modularity metric since Locatello et al.

Original: (2019) suggests that Modularity is inconsistent with other metrics.
Revised: (2019) suggests that it is inconsistent with other metrics.

Original: Speciﬁcally, we use BetaVAE,

Original: FactorVAE, Mutual Information Gap (MIG), SAP, and DCI metrics. Please refer to Appendix A.4 for details.
Revised: Please refer to Appendix A.4 for details including the calculation of mutual information.

Original: Reference Methods Most of the previous state-of-art disentangled feature representation learning methods are either built on VAE (Kingma & Welling, 2013) or
Revised: Reference Methods Most of the previous state-of-art disentangled feature representation learning methods are either built on VAE (Kingma & Welling, 2013) or

Original: GAN (Goodfellow et al., 2014).
Revised: GAN (Goodfellow et al., 2014).

Original: β -VAE (Higgins et al., 2016) introduces a hyper-parameter to adjust the KL constraint in VAE.
Revised: β -VAE (Higgins et al., 2016) introduces a hyper-parameter to adjust the KL constraint in VAE.

Original: FactorVAE (Kim & Mnih, 2018) and βTCVAE use adversarial training to reduce the correlations on different dimensions of the latent code.
Revised: FactorVAE (Kim & Mnih, 2018) and βTCVAE use adversarial training to reduce the correlations on different dimensions of the latent code.

Original: DIPVAE (Kumar et al., 2017) pull the posterior to a factorized prior.
Revised: DIPVAE (Kumar et al., 2017) pull the posterior to a factorized prior.

Original: Besides the VAE-based methods, we also investigate Generative Adversarial Networks (GANs).
Revised: Besides the VAE-based methods, we also investigate Generative Adversarial Networks (GANs).

Original: InfoGAN (Chen et al., 2016) encourages the latent code to have high mutual information with the generated image.
Revised: InfoGAN (Chen et al., 2016) encourages the latent code to have high mutual information with the generated image.

Original: IB-GAN (Jeon et al., 2018) and InforGAN-CR (Lin et al., 2020) add more constraints on InfoGAN to further promote disentanglement.
Revised: IB-GAN (Jeon et al., 2018) and InforGAN-CR (Lin et al., 2020) add more constraints on InfoGAN to further promote disentanglement.

Original: 5. U NDERSTANDING THE L EARNT R EPRESENTATION
Revised: 5. U NDERSTANDING THE L EARNT R EPRESENTATION

Original: In this part, we make the qualitative study of the disentanglement of representation learned by
Revised: In this part, we make the qualitative study of the disentanglement of representation learned by

Original: BYOL.
Revised: BYOL.

Original: Since contrastive methods are not generative models, it is hard to directly do factorcontrolled pixel-wise reconstruction for visualization.
Revised: Since contrastive methods are not generative models, it is hard to directly do factorcontrolled pixel-wise reconstruction for visualization.

Original: We thus turn to measure the mutual information between learned features and ground truth factors to study that.
Revised: We thus turn to measure the mutual information between learned features and ground truth factors to study that.

Original: The model is analyzed on dSprites dataset.
Revised: The model is analyzed on dSprites dataset.

Original: dSprites has in total ﬁve factors (shape, scale, orientation, position x, and position y).
Revised: dSprites has in total ﬁve factors (shape, scale, orientation, position x, and position y).

Original: But as explained in Appendix A.3, the orientation is ill-deﬁned with ambiguity.
Revised: But as explained in Appendix A.3, the orientation is ill-deﬁned with ambiguity.

Original: For example, it is impossible to distinguish if a square rotates 0 degrees or 180 degrees.
Revised: For example, it is impossible to distinguish if a square rotates 0 degrees or 180 degrees.

Original: Therefore, we ignore this factor temporarily in the following discussion.
Revised: Therefore, we ignore this factor temporarily in the following discussion.

Original: After encoding an input image to a representation vector, we compute the mutual information between each factor and each representation dimension.
Revised: After encoding an input image to a representation vector, we compute the mutual information between each factor and each representation dimension.

Original: The mutual information between all dimensions and all four left factors are included in Figure 3.
Revised: The mutual information between all dimensions and all four left factors are included in Figure 3.

Original: It shows an evident pattern that a ground truth factor can correspond to multiple representation dimensions but a single dimension of representation only has the high response to one factor.
Revised: It shows an evident pattern that a ground truth factor can correspond to multiple representation dimensions but a single dimension of representation only has the high response to one factor.

Original: To have a more intuitive understanding of to what extent a representation dimension may respond to more than one factor, we deﬁne metrics by the normalized co-occurrence of mutual information.
Revised: To have a more intuitive understanding of to what extent a representation dimension may respond to more than one factor, we deﬁne metrics by the normalized co-occurrence of mutual information.

Original: Given the mutual information between the representation vector and the i th factor, noted as M i ,
Revised: Given the mutual information between the representation vector and the i th factor, noted as M i ,

Original: M ji is the mutual information by the j th dimension.
Revised: M ji is the mutual information by the j th dimension.

Original: Then, the normalized co-occurrence of mutual information between the i th factor and the k th factor is
Revised: Then, the normalized co-occurrence of mutual information between the i th factor and the k th factor is

Original: [Equation]
Revised: [Equation]

Original: where L is the representation vector length.
Revised: where L is the representation vector length.

Original: The deﬁnition conforms to the InfoMax (Linsker, 1988) principle of maximizing the mutual information of conceptually correlated pairs.
Revised: The deﬁnition conforms to the InfoMax (Linsker, 1988) principle of maximizing the mutual information of conceptually correlated pairs.

Original: We visualized the normalized co-occurrence of mutual information among the four factors by the learned representation in Figure 4.
Revised: We visualized the normalized co-occurrence of mutual information among the four factors by the learned representation in Figure 4.

Original: It agrees with the pattern revealed in Figure 3 that representation code on a certain dimension will not have high mutual information to more than one factor.
Revised: It agrees with the pattern revealed in Figure 3 that representation code on a certain dimension will not have high mutual information to more than one factor.

Original: Moreover, it shows the independence degree of factor pairs.
Revised: Moreover, it shows the independence degree of factor pairs.

Original: For example, the shape and scale of dSprites objects are not fully disentangled because objects with the same scale value but in different shapes have different pixel area.
Revised: For example, the shape and scale of dSprites objects are not fully disentangled because objects with the same scale value but in different shapes have different pixel area.

Original: The co-occurrence of mutual information on them is thus slightly higher than other pairs.
Revised: The co-occurrence of mutual information on them is thus slightly higher than other pairs.

Original: [Figure]
Revised: [Figure]

Original: The analysis above recognizes the pattern of “group disentanglement”.
Revised: The analysis above recognizes the pattern of “group disentanglement”.

Original: Then it comes a natural question of whether we can get a more compact and interpretable representation.
Revised: Then it comes a natural question of whether we can get a more compact and interpretable representation.

Original: Here, we reduce the representation dimension by PCA to 10.
Revised: Here, we reduce the representation dimension by PCA to 10.

Original: Figure 5 shows the result of latent variation when changing only one factor at once.
Revised: Figure 5 shows the result of latent variation when changing only one factor at once.

Original: Given three images with only one factor’s value different, we generate the 10-dim representation vectors from them.
Revised: Given three images with only one factor’s value different, we generate the 10-dim representation vectors from them.

Original: Then, we compute the variance across the three vectors.
Revised: Then, we compute the variance across the three vectors.

Original: The Figure 5(a) and (b) show how reduced latent code changes when manipulating position x and position y factor respectively.
Revised: The Figure 5(a) and (b) show how reduced latent code changes when manipulating position x and position y factor respectively.

Original: It shows good disentanglement that only one representation dimension has high variation.
Revised: It shows good disentanglement that only one representation dimension has high variation.

Original: However, in Figure 5(c) we show a failure mode of the ill-posed factor orientation that change of factor causes both 6th and 9th dimension of reduced representation to have large variations.
Revised: However, in Figure 5(c) we show a failure mode of the ill-posed factor orientation that change of factor causes both 6th and 9th dimension of reduced representation to have large variations.

Original: From the results, we observe that manipulating one well-deﬁned independent factor causes evident variance on only one dimension.
Revised: From the results, we observe that manipulating one well-deﬁned independent factor causes evident variance on only one dimension.

Original: Compared with the original group disentanglement pattern, the dimension-reduced version becomes more compact and interpretable.
Revised: Compared with the original group disentanglement pattern, the dimension-reduced version becomes more compact and interpretable.

Original: To summarize, we have found strong empirical evidence about the “group disentanglement” property of the representation learned by contrastive learning.
Revised: To summarize, we have found strong empirical evidence about the “group disentanglement” property of the representation learned by contrastive learning.

Original: Moreover, we have shown that the representation can be made more compact through proper unsupervised reduction.
Revised: Moreover, we have shown that the representation can be made more compact through proper unsupervised reduction.

Original: 5. C OMPARISON ON E XISTING B ENCHMARKS
Revised: 5. C OMPARISON ON E XISTING B ENCHMARKS

Original: To quantitatively evaluate the disentanglement of contrastive learning, we compare contrastive learning with the previous line of disentangled representation learning methods on commonly used benchmarks in this section.
Revised: To quantitatively evaluate the disentanglement of contrastive learning, we compare contrastive learning with the previous line of disentangled representation learning methods on commonly used benchmarks in this section.

Original: In the previous works, a lot of datasets have been proposed to evaluate the disentanglement learning method.
Revised: In the previous works, a lot of datasets have been proposed to evaluate the disentanglement learning method.

Original: Locatello et al.
Revised: Locatello et al.

Original: (2019) suggests that new method should be evaluated on a wide range of datasets to minimize potential dataset biases.
Revised: (2019) suggests that new method should be evaluated on a wide range of datasets to minimize potential dataset biases.

Original: In this part, we follow the evaluation protocol of Locatello et al.
Revised: In this part, we follow the evaluation protocol of Locatello et al.

Original: (2019) to gain quantitative disentanglement measurement on dSprites dataset.
Revised: (2019) to gain quantitative disentanglement measurement on dSprites dataset.

Original: For results on SmallNORB, Cars3D and Shapes3D, please refer to the results in Appendix.
Revised: For results on SmallNORB, Cars3D and Shapes3D, please refer to the results in Appendix.

Original: B. The results of VAEs are reported based on the best conﬁguration from Locatello et al.
Revised: B. The results of VAEs are reported based on the best conﬁguration from Locatello et al.

Original: We calculate the mean performance and standard deviation from the released raw logs.
Revised: We calculate the mean performance and standard deviation from the released raw logs.

Original: The results of InfoGAN, IB-GAN and InfoGAN-CR are reported in Lin et al.
Revised: The results of InfoGAN, IB-GAN and InfoGAN-CR are reported in Lin et al.

Original: and we use them its ofﬁcially released checkpoint to evaluate under our protocol.
Revised: and we use them its ofﬁcially released checkpoint to evaluate under our protocol.

Original: The performance of Ada-GVAE and Ada-ML-VAE are reported in a previous paper (Locatello et al., 2020).
Revised: The performance of Ada-GVAE and Ada-ML-VAE are reported in a previous paper (Locatello et al., 2020).

Original: In the independent component analysis community, we can also ﬁnd some methods to compare with but usually they are designed for different benchmarks and metrics.
Revised: In the independent component analysis community, we can also ﬁnd some methods to compare with but usually they are designed for different benchmarks and metrics.

Original: We adopt the method reported in ICE-BeeM (Khemakhem et al., 2020b) and migrate it to the benchmarks we evaluate on.
Revised: We adopt the method reported in ICE-BeeM (Khemakhem et al., 2020b) and migrate it to the benchmarks we evaluate on.

Original: However, as we focus on unsupervised learning setting, no factor attribute is available for training data, so the conditional denoising score matching (CDSM) version of ICE-BeeM can not be used here.
Revised: However, as we focus on unsupervised learning setting, no factor attribute is available for training data, so the conditional denoising score matching (CDSM) version of ICE-BeeM can not be used here.

Original: We turn to use its unconditional version, and we term it as EBM (energy-based model) in following content.
Revised: We turn to use its unconditional version, and we term it as EBM (energy-based model) in following content.

Original: [Table]
Revised: [Table]

Original: It shows strong and robust disentanglement property of BYOL.
Revised: It shows strong and robust disentanglement property of BYOL.

Original: All evaluation settings follow the conﬁguration of Locatello et al.
Revised: All evaluation settings follow the conﬁguration of Locatello et al.

Original: The results strongly suggest that contrastive learning can well promote disentangled features.
Revised: The results strongly suggest that contrastive learning can well promote disentangled features.

Original: Moreover, it shows the evidence that different metrics can not be precisely aligned as they use different perspectives to measure the disentanglement degree, which itself evens lacks a uniﬁed deﬁnition.
Revised: Moreover, it shows the evidence that different metrics can not be precisely aligned as they use different perspectives to measure the disentanglement degree, which itself evens lacks a uniﬁed deﬁnition.

Original: Moreover, as the latent dimension of VAE-based methods is only 128, smaller than that of BYOL (1000-d), we increase the representation dimension of VAEs to 1000 for a more fair comparison.
Revised: Moreover, as the latent dimension of VAE-based methods is only 128, smaller than that of BYOL (1000-d), we increase the representation dimension of VAEs to 1000 for a more fair comparison.

Original: The results are in Table 11 in Appendix, showing that increasing latent dimension does not boost the VAEs’ disentanglement.
Revised: The results are in Table 11 in Appendix, showing that increasing latent dimension does not boost the VAEs’ disentanglement.

Original: Therefore, the performance gap between BYOL and VAE-based methods does not come from the change of representation dimension.
Revised: Therefore, the performance gap between BYOL and VAE-based methods does not come from the change of representation dimension.

Original: 5. C OMPARISON ON R EAL -W ORLD D ATASETS
Revised: 5. C OMPARISON ON R EAL -W ORLD D ATASETS

Original: Although the four datasets above can establish how well each method performs, those datasets are all synthetic and are limited in various aspects.
Revised: Although the four datasets above can establish how well each method performs, those datasets are all synthetic and are limited in various aspects.

Original: For example, dSprites only has one object per image on a black background; SmallNORB contains synthetic images of 3D objects viewing from different angles, but only contains 50 instances within 5 categories.
Revised: For example, dSprites only has one object per image on a black background; SmallNORB contains synthetic images of 3D objects viewing from different angles, but only contains 50 instances within 5 categories.

Original: High performance on those datasets does not necessarily transfer to real world images, where there might be a lot more variations and unseen objects during test time.
Revised: High performance on those datasets does not necessarily transfer to real world images, where there might be a lot more variations and unseen objects during test time.

Original: To get a clearer sense of how each of the methods would perform on the real world datasets, we further evaluate our method and several other methods on CelebA.
Revised: To get a clearer sense of how each of the methods would perform on the real world datasets, we further evaluate our method and several other methods on CelebA.

Original: For evaluation of MIG and SAP metrics, to make a fair comparison, the representation vector of all methods is reduced to 40 dimensions.
Revised: For evaluation of MIG and SAP metrics, to make a fair comparison, the representation vector of all methods is reduced to 40 dimensions.

Original: The results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: BYOL continues to show high disentanglement compared with other methods.
Revised: BYOL continues to show high disentanglement compared with other methods.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E FFECTS OF H YPER -P ARAMS
Revised: E FFECTS OF H YPER -P ARAMS

Original: Hyper-parameters signiﬁcantly affect the performance of contrastive learning.
Revised: Hyper-parameters signiﬁcantly affect the performance of contrastive learning.

Original: In this section, we systematically study the inﬂuence on disentanglement property from some recognized crucial inductive bias for contrastive learning.
Revised: In this section, we systematically study the inﬂuence on disentanglement property from some recognized crucial inductive bias for contrastive learning.

Original: The batch normalization in BYOL has been considered the reason why BYOL does not collapse even without negative pairs for long.
Revised: The batch normalization in BYOL has been considered the reason why BYOL does not collapse even without negative pairs for long.

Original: However, a previous work (Richemond et al., 2020) ﬁnds BYOL can avoid collapsing even without batch statistics.
Revised: However, a previous work (Richemond et al., 2020) ﬁnds BYOL can avoid collapsing even without batch statistics.

Original: But the normalization strategy between layers is still recognized as a key variant of BYOL.
Revised: But the normalization strategy between layers is still recognized as a key variant of BYOL.

Original: Thus we experiment with ﬁve normalization layers conﬁguration in the encoder network on the dSprites dataset.
Revised: Thus we experiment with ﬁve normalization layers conﬁguration in the encoder network on the dSprites dataset.

Original: The results are shown in Table 3.
Revised: The results are shown in Table 3.

Original: We ﬁnd the commonly used BN decreases the disentanglement performances.
Revised: We ﬁnd the commonly used BN decreases the disentanglement performances.

Original: No normalization already does well regarding feature disentanglement.
Revised: No normalization already does well regarding feature disentanglement.

Original: Group norm (Wu & He, 2018) and layer norm (Ba et al., 2016) are the two best normalization techniques.
Revised: Group norm (Wu & He, 2018) and layer norm (Ba et al., 2016) are the two best normalization techniques.

Original: On dSprites, instance norm (Ulyanov et al., 2017) completely breaks the contrastive learning process.
Revised: On dSprites, instance norm (Ulyanov et al., 2017) completely breaks the contrastive learning process.

Original: We provide more study on normalization in Table 13 and Table 14 in Appendix where more variables, such as batch size, are taken into consideration.
Revised: We provide more study on normalization in Table 13 and Table 14 in Appendix where more variables, such as batch size, are taken into consideration.

Original: From all the studies, it remains unclear why batch norm could harm the feature disentanglement.
Revised: From all the studies, it remains unclear why batch norm could harm the feature disentanglement.

Original: However, this phenomenon was also empirically observed in the supervised learning literature as well.
Revised: However, this phenomenon was also empirically observed in the supervised learning literature as well.

Original: For example, in the Bau et al.
Revised: For example, in the Bau et al.

Original: (2017) paper, the author shows that CNN with batch norm layer has signiﬁcantly fewer interpretable concepts than those without BN.
Revised: (2017) paper, the author shows that CNN with batch norm layer has signiﬁcantly fewer interpretable concepts than those without BN.

Original: We also perform an ablation study on other hyper-parameters.
Revised: We also perform an ablation study on other hyper-parameters.

Original: For example, we ﬁnd the representation trained on one dataset can show transferrable disentanglement to other datasets and BN causes model collapse with a big learning rate (Table 12).
Revised: For example, we ﬁnd the representation trained on one dataset can show transferrable disentanglement to other datasets and BN causes model collapse with a big learning rate (Table 12).

Original: More details are provided in Appendix D.
Revised: More details are provided in Appendix D.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we show some empirical study on the disentanglement property by contrastive learning.
Revised: In this paper, we show some empirical study on the disentanglement property by contrastive learning.

Original: We ﬁnd with minor modiﬁcation, the existing BYOL framework can achieve state-of-the-art disentanglement performance on multiple benchmarks and under diverse quantitative metrics.
Revised: We ﬁnd with minor modiﬁcation, the existing BYOL framework can achieve state-of-the-art disentanglement performance on multiple benchmarks and under diverse quantitative metrics.

Original: Besides, the qualitative study reveals the different “disentanglement” nature of contrastive learning that the representation actually shows “group disentanglement”.
Revised: Besides, the qualitative study reveals the different “disentanglement” nature of contrastive learning that the representation actually shows “group disentanglement”.

Original: It is thus a relaxed form of disentanglement due to weaker bound of compactness.
Revised: It is thus a relaxed form of disentanglement due to weaker bound of compactness.

Original: To the best of our knowledge, we are the ﬁrst to build quantitative disentanglement benchmark with contrastive learning involved and the ﬁrst to build benchmark of disentanglement measurement on real-world datasets (CelebA).
Revised: To the best of our knowledge, we are the ﬁrst to build quantitative disentanglement benchmark with contrastive learning involved and the ﬁrst to build benchmark of disentanglement measurement on real-world datasets (CelebA).

Original: Recently, the study of contratsive learning, or generally self-supervised learning, is still motivated by empirical observations.
Revised: Recently, the study of contratsive learning, or generally self-supervised learning, is still motivated by empirical observations.

Original: We wish our work can reveal some clues on this line of study.
Revised: We wish our work can reveal some clues on this line of study.

Original:  R EPRODUCIBILITY
Revised:  R EPRODUCIBILITY

Original: In this section, we provide the information required to reproduce our results reported in the main text.
Revised: In this section, we provide the information required to reproduce our results reported in the main text.

Original: And we commit to making the code implementation and evaluating checkpoints public.
Revised: And we commit to making the code implementation and evaluating checkpoints public.

Original: BYOL implementation For the implementation details of BYOL, please refer to Appendix A.1.
Revised: BYOL implementation For the implementation details of BYOL, please refer to Appendix A.1.

Original: The model architecture, training setups, and dataset preprocessing are all explained in detail.
Revised: The model architecture, training setups, and dataset preprocessing are all explained in detail.

Original: VAE methods implementation For evaluation on synthetic datasets, i.e., dSprites, Cars3D,
Revised: VAE methods implementation For evaluation on synthetic datasets, i.e., dSprites, Cars3D,

Original: SmallNORB and Shapes3D, the disentanglement score is from the original logs of disentanglement lib Locatello et al.
Revised: SmallNORB and Shapes3D, the disentanglement score is from the original logs of disentanglement lib Locatello et al.

Original: In the released logs, each method has different training conﬁgurations, and our reported result is from the conﬁguration with the highest average performance overall provided random seeds.
Revised: In the released logs, each method has different training conﬁgurations, and our reported result is from the conﬁguration with the highest average performance overall provided random seeds.

Original: For evaluation on CelebA dataset, we follow an open-sourced implementation in Pytorch 2 and align the encoder architecture of all methods to be the same as described in Appendix A.1.
Revised: For evaluation on CelebA dataset, we follow an open-sourced implementation in Pytorch 2 and align the encoder architecture of all methods to be the same as described in Appendix A.1.

Original: Parameters are kept as the default well-tuned version in the provided implementation.
Revised: Parameters are kept as the default well-tuned version in the provided implementation.

Original: GAN methods implementation It is hard to insert GAN methods’ performance in the benchmark as the training is not always stable and the discriminator weights are usually not provided in many public codebases.
Revised: GAN methods implementation It is hard to insert GAN methods’ performance in the benchmark as the training is not always stable and the discriminator weights are usually not provided in many public codebases.

Original: When evaluating on synthetic datasets, the FactorVAE scores of InforGAN, IBGAN, and InfoGAN-CR are provided in the paper of Lin et al.
Revised: When evaluating on synthetic datasets, the FactorVAE scores of InforGAN, IBGAN, and InfoGAN-CR are provided in the paper of Lin et al.

Original: But the evaluation of other metrics in Lin et al.
Revised: But the evaluation of other metrics in Lin et al.

Original: (2020) uses a not aligned settings with Locatello et al.
Revised: (2020) uses a not aligned settings with Locatello et al.

Original: , so we check its ofﬁcially release 3 to reevaluate the provided implementation and model weights under the uniﬁed evaluation setup.
Revised: , so we check its ofﬁcially release 3 to reevaluate the provided implementation and model weights under the uniﬁed evaluation setup.

Original: We perform the same evaluation process for results on CelebA dataset.
Revised: We perform the same evaluation process for results on CelebA dataset.

Original: Energy-based Model (EBM)
Revised: Energy-based Model (EBM)

Original: We refer to the implementation of ICE-BeeM (Khemakhem et al., 2020b) for this method.
Revised: We refer to the implementation of ICE-BeeM (Khemakhem et al., 2020b) for this method.

Original: We use the ofﬁcially released codebase for it 4 .
Revised: We use the ofﬁcially released codebase for it 4 .

Original: The encoder implementation has been aligned with our default already.
Revised: The encoder implementation has been aligned with our default already.

Original: The only modiﬁcation we make is to use the unconditional version instead of its default conditional version in loss computation.
Revised: The only modiﬁcation we make is to use the unconditional version instead of its default conditional version in loss computation.

Original: Please refer to the runners/real data runner.py ﬁle for details.
Revised: Please refer to the runners/real data runner.py ﬁle for details.

Original: Besides, we add the dataset deﬁnition for dSprites,
Revised: Besides, we add the dataset deﬁnition for dSprites,

Original: SmallNORB, Cars3D, and Shapes3D to it as the implementation for our method.
Revised: SmallNORB, Cars3D, and Shapes3D to it as the implementation for our method.

Original: All hyperparameters are kept the default in its released version.
Revised: All hyperparameters are kept the default in its released version.

Original: Evaluation Protocol Though our study is focused on disentanglement property, the evaluation of a representation model is two-step.
Revised: Evaluation Protocol Though our study is focused on disentanglement property, the evaluation of a representation model is two-step.

Original: In the ﬁrst step, we perform factor prediction as a downstream task based on the learned representation model.
Revised: In the ﬁrst step, we perform factor prediction as a downstream task based on the learned representation model.

Original: Only when the accuracy of factor prediction is high enough to ensure the model weights “valid”, we would go to the evaluation of disentanglement metric scores.
Revised: Only when the accuracy of factor prediction is high enough to ensure the model weights “valid”, we would go to the evaluation of disentanglement metric scores.

Original: For the implementation details of factor prediction, please refer to Appendix A.3.
Revised: For the implementation details of factor prediction, please refer to Appendix A.3.

Original: For disentanglement metrics evaluation, we use the ofﬁcial implementation of Locatello et al.
Revised: For disentanglement metrics evaluation, we use the ofﬁcial implementation of Locatello et al.

Original: (2019), i.e. disentanglement lib .
Revised: (2019), i.e. disentanglement lib .

Original: The settings of some important parameters in Appendix A.4.
Revised: The settings of some important parameters in Appendix A.4.

Original: A A PPENDIX A. I MPLEMENTATION OF BYOL MODEL
Revised: A A PPENDIX A. I MPLEMENTATION OF BYOL MODEL

Original: Architecture We describe the implementation details of BYOL in this part.
Revised: Architecture We describe the implementation details of BYOL in this part.

Original: To make a fair comparison with previous methods, for synthetic datasets, i.e. dSprites, Cars3D, SmallNORB, and
Revised: To make a fair comparison with previous methods, for synthetic datasets, i.e. dSprites, Cars3D, SmallNORB, and

Original: Shapes3D, we follow the encoder architecture in Factor VAE (Kim & Mnih, 2018).
Revised: Shapes3D, we follow the encoder architecture in Factor VAE (Kim & Mnih, 2018).

Original: The pipeline details are shown in Table 4.
Revised: The pipeline details are shown in Table 4.

Original: After each shown convolutional layer in the ﬁgure, there is a ReLU activation layer.
Revised: After each shown convolutional layer in the ﬁgure, there is a ReLU activation layer.

Original: By default, there is also a group normalization (group number = 4) layer.
Revised: By default, there is also a group normalization (group number = 4) layer.

Original: So, the encoder is a stack of (Conv-ReLU-GN) blocks.
Revised: So, the encoder is a stack of (Conv-ReLU-GN) blocks.

Original: For CelebA dataset, the encoder is the commonly adopted ResNet-50 (He et al., 2016) backbone.
Revised: For CelebA dataset, the encoder is the commonly adopted ResNet-50 (He et al., 2016) backbone.

Original: By default, the ﬁnal output channel number is 1000, i.e, D = 1000 .
Revised: By default, the ﬁnal output channel number is 1000, i.e, D = 1000 .

Revised: We note here that the group normalization we use as default has nothing necessarily connected with the group disentanglement property we study.

Original: Besides the representation network (encoder), BYOL also has a projector network and a predictor network.
Revised: Besides the representation network (encoder), BYOL also has a projector network and a predictor network.

Original: Both of them consists of a pipeline “Linear −→ BN −→ ReLU −→ Linear”.
Revised: Both of them consists of a pipeline “Linear −→ BN −→ ReLU −→ Linear”.

Original: The projection dimension is 256 and the hidden dimension of the projector is 4096.
Revised: The projection dimension is 256 and the hidden dimension of the projector is 4096.

Original: The predictor keeps a 256-dimensional feature vector in its pipeline.
Revised: The predictor keeps a 256-dimensional feature vector in its pipeline.

Revised: Encoder of BYOL

Original: [Table]
Revised: [Table]

Revised: (Conv-ReLU-Norm) blocks.

Original: Training settings We make minor modiﬁcation on the training setting of BYOL.
Revised: Training settings We make minor modiﬁcation on the training setting of BYOL.

Original: For training on all datasets, the images are resized to 64x64.
Revised: For training on all datasets, the images are resized to 64x64.

Original: For the data preprocessing, we copy 1-channel images of dSprites and SmallNORB to 3-channel.
Revised: For the data preprocessing, we copy 1-channel images of dSprites and SmallNORB to 3-channel.

Original: During training stage, we use such a pipeline of augmentation (in PyTorch -style):
Revised: During training stage, we use such a pipeline of augmentation (in PyTorch -style):

Original: RandomApply(transforms.
Revised: RandomApply(transforms.

Original: ColorJitter(0.8, 0.8, 0.8, 0.2), p=0.3) 2. transforms.
Revised: ColorJitter(0.8, 0.8, 0.8, 0.2), p=0.3) 2. transforms.

Original: RandomHorizontalFlip() 3.
Revised: RandomHorizontalFlip() 3.

Original: RandomApply(transforms.
Revised: RandomApply(transforms.

Original: GaussianBlur((3,3), (1.0, 2.0)), p=0.2) 4. transforms.
Revised: GaussianBlur((3,3), (1.0, 2.0)), p=0.2) 4. transforms.

Original: RandomResizeCrop(size=(64, 64), scale=(0.6,1.0)) 5.
Revised: RandomResizeCrop(size=(64, 64), scale=(0.6,1.0)) 5.

Original: normalization
Revised: normalization

Original: For dSprites and SmallNORB, image pixel value is uniformly normalized from [0,255] to [0,1.0].
Revised: For dSprites and SmallNORB, image pixel value is uniformly normalized from [0,255] to [0,1.0].

Original: For Cars3D, Shapes3D, and CelebA, we adopt the commonly used Imagenet-statistic normalization for preprocessing the image values.
Revised: For Cars3D, Shapes3D, and CelebA, we adopt the commonly used Imagenet-statistic normalization for preprocessing the image values.

Original: During training, we use Adam optimizer by default, whose learning rate is 3 e − 4 without weight decay.
Revised: During training, we use Adam optimizer by default, whose learning rate is 3 e − 4 without weight decay.

Original: The batch size is set to be 512 without exceptional notation.
Revised: The batch size is set to be 512 without exceptional notation.

Original: For evaluation on dSprites,
Revised: For evaluation on dSprites,

Original: Shapes3D and CelebA, we select the weights after training for 15 epochs for evaluation.
Revised: Shapes3D and CelebA, we select the weights after training for 15 epochs for evaluation.

Original: We select the weights after training for 140 epochs for evaluation on Cars3D and weights of the 200th epoch on SmallNORB considering the small scale of these two datasets.
Revised: We select the weights after training for 140 epochs for evaluation on Cars3D and weights of the 200th epoch on SmallNORB considering the small scale of these two datasets.

Original: As explained in some ablation experiments, the default training setting is not the one we could get the highest disentanglement score.
Revised: As explained in some ablation experiments, the default training setting is not the one we could get the highest disentanglement score.

Original: For example, using Group Normalization + Weight Standardization achieves a higher disentanglement score than only using Group Normalization or not using normalization.
Revised: For example, using Group Normalization + Weight Standardization achieves a higher disentanglement score than only using Group Normalization or not using normalization.

Original: But we aim to make minor modiﬁcations to the original BYOL implementation and follow previously commonly used setups.
Revised: But we aim to make minor modiﬁcations to the original BYOL implementation and follow previously commonly used setups.

Original: To decrease the inﬂuence of randomness, we train each model conﬁguration multiple times with different random seeds.
Revised: To decrease the inﬂuence of randomness, we train each model conﬁguration multiple times with different random seeds.

Original: We report the results by the average and three random seeds.
Revised: We report the results by the average and three random seeds.

Original: To be precise, as our implementation is based on Pytorch, we would set initialize the libraries of numpy , torch , torch.cuda and random with the same random seeds.
Revised: To be precise, as our implementation is based on Pytorch, we would set initialize the libraries of numpy , torch , torch.cuda and random with the same random seeds.

Original: We use 0, 1, 2 as the three values of random seeds in trails.
Revised: We use 0, 1, 2 as the three values of random seeds in trails.

Original: A. I MPLEMENTATION OF VAE MODELS ON C ELEB A
Revised: A. I MPLEMENTATION OF VAE MODELS ON C ELEB A

Original: The benchmark (Locatello et al., 2019) provides VAE-based methods’ disentanglement performance on synthetics datasets only, e.g., dSprites, Shapes3D, Cars3D, and SmallNORB.
Revised: The benchmark (Locatello et al., 2019) provides VAE-based methods’ disentanglement performance on synthetics datasets only, e.g., dSprites, Shapes3D, Cars3D, and SmallNORB.

Original: For the real-world
Revised: For the real-world

Original: CelebA dataset, we need to train the models by ourselves to gain the results in Table 2.
Revised: CelebA dataset, we need to train the models by ourselves to gain the results in Table 2.

Original: We use an open-sourced implementation 5 for VanillaVAE, β -VAE, Factor-VAE, β -TCVAE and DIP-VAE-I.
Revised: We use an open-sourced implementation 5 for VanillaVAE, β -VAE, Factor-VAE, β -TCVAE and DIP-VAE-I.

Original: We follow the provided well-tuned hyper-parameters provided by them.
Revised: We follow the provided well-tuned hyper-parameters provided by them.

Original: Please refer to the repository for details.
Revised: Please refer to the repository for details.

Original: Linear classiﬁer for factor prediction
Revised: Linear classiﬁer for factor prediction

Original: [Table]
Revised: [Table]

Original: , as each factor has n k (1 ≤ k ≤ K ) values, we have K linear layers following the last shared layer in parallel.
Revised: , as each factor has n k (1 ≤ k ≤ K ) values, we have K linear layers following the last shared layer in parallel.

Original: These layers predict the factor value on the corresponding K factors.
Revised: These layers predict the factor value on the corresponding K factors.

Original: A. I MPLEMENTATION OF F ACTOR P REDICTION
Revised: A. I MPLEMENTATION OF F ACTOR P REDICTION

Original: As mentioned in the main paper, for each gained representation model, i.e., encoder, we will ﬁrst conﬁrm it “valid” by training a linear classiﬁer for factor prediction.
Revised: As mentioned in the main paper, for each gained representation model, i.e., encoder, we will ﬁrst conﬁrm it “valid” by training a linear classiﬁer for factor prediction.

Original: In the stage, the encoder weights will be frozen and only the attached classiﬁer trained.
Revised: In the stage, the encoder weights will be frozen and only the attached classiﬁer trained.

Original: For the task on all datasets, we use a uniﬁed classiﬁer as shown in Table 5.
Revised: For the task on all datasets, we use a uniﬁed classiﬁer as shown in Table 5.

Original: The classiﬁer has stacked fully connected layers (linear layers) and multiple in-parallel heads for factor prediction.
Revised: The classiﬁer has stacked fully connected layers (linear layers) and multiple in-parallel heads for factor prediction.

Original: On different target datasets, the number of factors and the possible values of factors vary.
Revised: On different target datasets, the number of factors and the possible values of factors vary.

Original: For the datasets we evaluate on, the value range and the accuracy threshold to conﬁrm the quality of a representation acceptable are shown in Table 6.
Revised: For the datasets we evaluate on, the value range and the accuracy threshold to conﬁrm the quality of a representation acceptable are shown in Table 6.

Original: We use a well-trained BetaVAE model weights to do the factor prediction ﬁrst.
Revised: We use a well-trained BetaVAE model weights to do the factor prediction ﬁrst.

Original: Its prediction accuracy serves as the threshold in this sanity check stage.
Revised: Its prediction accuracy serves as the threshold in this sanity check stage.

Original: Usually, the accuracy threshold is set to be 0.8 or higher.
Revised: Usually, the accuracy threshold is set to be 0.8 or higher.

Original: But for the orientation factor on dSprites and Shapes3D, we relax the constraint to 0.5.
Revised: But for the orientation factor on dSprites and Shapes3D, we relax the constraint to 0.5.

Original: The reason is that the factors are ill-deﬁned to have ambiguity.
Revised: The reason is that the factors are ill-deﬁned to have ambiguity.

Original: For example, when a square or ellipse object in dSprites has an orientation of 180 degrees or 0 degrees, the image can be the same while the factors of orientation are regarded differently.
Revised: For example, when a square or ellipse object in dSprites has an orientation of 180 degrees or 0 degrees, the image can be the same while the factors of orientation are regarded differently.

Original: For this reason, these factors bring noise into our study and validate the model weights.
Revised: For this reason, these factors bring noise into our study and validate the model weights.

Original: We, therefore, relax the accuracy requirement for them in practice.
Revised: We, therefore, relax the accuracy requirement for them in practice.

Original: Besides, SmallNORB is the most difﬁcult dataset by our observation.
Revised: Besides, SmallNORB is the most difﬁcult dataset by our observation.

Original: The factor prediction accuracy on it is usually much lower, especially for azimuth and elevation, both of which are even harder to be predicted.
Revised: The factor prediction accuracy on it is usually much lower, especially for azimuth and elevation, both of which are even harder to be predicted.

Original: [Table]
Revised: [Table]

Original: A. E VALUATION M ETRICS
Revised: A. E VALUATION M ETRICS

Original: Five metrics are used to quantify the disentanglement of our model.
Revised: Five metrics are used to quantify the disentanglement of our model.

Original: Beta Vae Metrics Introduced in Higgins et al.
Revised: Beta Vae Metrics Introduced in Higgins et al.

Original: , beta vae metrics assumes each dimension corresponds to one category in linear classiﬁer.
Revised: , beta vae metrics assumes each dimension corresponds to one category in linear classiﬁer.

Original: Representations are obtained after generated samples with only one factor k ﬁxed.
Revised: Representations are obtained after generated samples with only one factor k ﬁxed.

Original: Calculating the summation of the divergence between different representations and putting this result into a linear classiﬁer, we train a model that possibly outputs the corresponding k.
Revised: Calculating the summation of the divergence between different representations and putting this result into a linear classiﬁer, we train a model that possibly outputs the corresponding k.

Original: The accuracy of this linear model is the value of beta vae metrics.
Revised: The accuracy of this linear model is the value of beta vae metrics.

Original: Factor Vae Metrics Kim & Mnih (2018) argues the beta vae metrics has the tendency to fail into a spurious disentanglement, and proposes a new metrics based on a linear classiﬁer.
Revised: Factor Vae Metrics Kim & Mnih (2018) argues the beta vae metrics has the tendency to fail into a spurious disentanglement, and proposes a new metrics based on a linear classiﬁer.

Original: Representations are obtained after generated samples with only factor k ﬁxed.
Revised: Representations are obtained after generated samples with only factor k ﬁxed.

Original: Normalizing each dimension in representations in terms of standard deviation.
Revised: Normalizing each dimension in representations in terms of standard deviation.

Original: Index of dimension with lowest variances of normalized representation and the factor index k is the input/output for the linear classiﬁer.
Revised: Index of dimension with lowest variances of normalized representation and the factor index k is the input/output for the linear classiﬁer.

Original: The accuracy of this linear model is the value of factor vae metrics.
Revised: The accuracy of this linear model is the value of factor vae metrics.

Original: Mutual Information Gap Chen et al.
Revised: Mutual Information Gap Chen et al.

Original: (2018) assumes the disentanglement model has the property that most information of one speciﬁc factor is contained in one dimension or a group of certain dimensions.
Revised: (2018) assumes the disentanglement model has the property that most information of one speciﬁc factor is contained in one dimension or a group of certain dimensions.

Original: The mutual information gap is the summation of the difference of highest and secondhighest normalized mutual information between ﬁxed k factor and dimensions in representations as shown below:
Revised: The mutual information gap is the summation of the difference of highest and secondhighest normalized mutual information between ﬁxed k factor and dimensions in representations as shown below:

Original: [Equation]
Revised: [Equation]

Original: Where K is the overall number of ground truth factors, z k is the factors of latent variables and j k = arg max j I ( v j , z k ) .
Revised: Where K is the overall number of ground truth factors, z k is the factors of latent variables and j k = arg max j I ( v j , z k ) .

Original: DCI disentanglement As Eastwood & Williams (2018) suggests, disentanglement is the entropy of the relative importance, evaluated with a Lasso or a Random Forest classiﬁer, of each dimension for speciﬁc factors.
Revised: DCI disentanglement As Eastwood & Williams (2018) suggests, disentanglement is the entropy of the relative importance, evaluated with a Lasso or a Random Forest classiﬁer, of each dimension for speciﬁc factors.

Original: Completeness is the entropy of the possibility that one factor is captured by a single code variable.
Revised: Completeness is the entropy of the possibility that one factor is captured by a single code variable.

Original: Informativeness, overlapping with Disentanglement, is the prediction error of the model predicting factors.
Revised: Informativeness, overlapping with Disentanglement, is the prediction error of the model predicting factors.

Original: SAP Kumar et al.
Revised: SAP Kumar et al.

Original: (2017) proposes the Separated Attribute Predictability (SAP) score.
Revised: (2017) proposes the Separated Attribute Predictability (SAP) score.

Original: A score metrics is computed with classiﬁcation score of predicting j th factors on i th dimension as the ij th entry.
Revised: A score metrics is computed with classiﬁcation score of predicting j th factors on i th dimension as the ij th entry.

Original: SAP is the mean of the difference of the highest and second-highest scores for each column.
Revised: SAP is the mean of the difference of the highest and second-highest scores for each column.

Original: For evaluation protocol, we follow the implementation provided by Locatello et al.
Revised: For evaluation protocol, we follow the implementation provided by Locatello et al.

Original: Despite exception, the evaluation batch size is 64, the prune dims.threshold is 0.06.
Revised: Despite exception, the evaluation batch size is 64, the prune dims.threshold is 0.06.

Original: If a classiﬁer is required to be trained during evaluation, num train is 10000 and num eval is 5000.
Revised: If a classiﬁer is required to be trained during evaluation, num train is 10000 and num eval is 5000.

Original: For Mutual information computation, the used discretizer function is the histogram discretizer and the number of bins in discretization is 20.
Revised: For Mutual information computation, the used discretizer function is the histogram discretizer and the number of bins in discretization is 20.

Original: For evaluation of MIG and SAP on dSprites, SmallNORB, Cars3D, and Shapes3D, BYOL representation vectors are reduced to 10 dimensions by PCA to be aligned with other methods.
Revised: For evaluation of MIG and SAP on dSprites, SmallNORB, Cars3D, and Shapes3D, BYOL representation vectors are reduced to 10 dimensions by PCA to be aligned with other methods.

Original: For evaluation of MIG and SAP on CelebA
Revised: For evaluation of MIG and SAP on CelebA

Original: [Table]
Revised: [Table]

Original: B M ORE BENCHMARK RESULTS
Revised: B M ORE BENCHMARK RESULTS

Original: We report the benchmark results on dSprites in the main text (Table 1).
Revised: We report the benchmark results on dSprites in the main text (Table 1).

Original: We continue to report the full benchmark results on other datasets in the Locatello et al.
Revised: We continue to report the full benchmark results on other datasets in the Locatello et al.

Original: (2019) evaluation protocol, i.e., Cars3D,
Revised: (2019) evaluation protocol, i.e., Cars3D,

Original: SmallNORB, and Shapes3D.
Revised: SmallNORB, and Shapes3D.

Original: As the large-scale benchmark of Locatello et al.
Revised: As the large-scale benchmark of Locatello et al.

Original: (2019) provides the original logs on Cars3D, and
Revised: (2019) provides the original logs on Cars3D, and

Original: SmallNORB datasets, we simply report the best conﬁguration VAE-based methods trained by them.
Revised: SmallNORB datasets, we simply report the best conﬁguration VAE-based methods trained by them.

Original: The original logs on Shapes3D are not available, so we train and evaluate on Shapes3D by ourselves and the output results are well aligned with the reported result in Locatello et al.
Revised: The original logs on Shapes3D are not available, so we train and evaluate on Shapes3D by ourselves and the output results are well aligned with the reported result in Locatello et al.

Original: The results of other methods come from the same resources as those on the dSprites benchmark.
Revised: The results of other methods come from the same resources as those on the dSprites benchmark.

Original: As we ﬁnd the energy-based model (EMB) implemented from ICE-BeeM (Khemakhem et al., 2020b) fails to learn valid representation on Cars3D with its default conﬁguration and hyperparameter settings, we do not report it here.
Revised: As we ﬁnd the energy-based model (EMB) implemented from ICE-BeeM (Khemakhem et al., 2020b) fails to learn valid representation on Cars3D with its default conﬁguration and hyperparameter settings, we do not report it here.

Original: By default, we report mean scores averaged on three random seeds for our method.
Revised: By default, we report mean scores averaged on three random seeds for our method.

Original: But on Shapes3D, as the mean scores for some methods are not available, we report the median performance instead as the default in Locatello et al.
Revised: But on Shapes3D, as the mean scores for some methods are not available, we report the median performance instead as the default in Locatello et al.

Original: The evaluation protocol is the same as that for dSprites as described in Appexdix A.4.
Revised: The evaluation protocol is the same as that for dSprites as described in Appexdix A.4.

Original: The full results are shown in Table 7, Table 8 and Table 9 respectively.
Revised: The full results are shown in Table 7, Table 8 and Table 9 respectively.

Original: On SmallNORB and Cars3D datasets, our method still achieves the state-of-the-art performance in terms of BetaVAE score, FactorVAE score, and DCI.
Revised: On SmallNORB and Cars3D datasets, our method still achieves the state-of-the-art performance in terms of BetaVAE score, FactorVAE score, and DCI.

Original: But its MIG and SAP results are outperformed by other methods.
Revised: But its MIG and SAP results are outperformed by other methods.

Original: This is because the BYOL approach learns a group-disentangled representation.
Revised: This is because the BYOL approach learns a group-disentangled representation.

Original: Therefore, there are multiple dimensions in the representation denoting the same ground truth factor.
Revised: Therefore, there are multiple dimensions in the representation denoting the same ground truth factor.

Original: MIG and SAP score computes the mutual information difference between the most related and second related dimension to each ground truth factor.
Revised: MIG and SAP score computes the mutual information difference between the most related and second related dimension to each ground truth factor.

Original: Thus a group disentangled feature can have a low MIG and SAP score, even after the dimension reduction.
Revised: Thus a group disentangled feature can have a low MIG and SAP score, even after the dimension reduction.

Original: On the other hand, on Shapes3D, our method fails to make the state-of-the-art performance on all metrics and the gap between its performance and others on
Revised: On the other hand, on Shapes3D, our method fails to make the state-of-the-art performance on all metrics and the gap between its performance and others on

Original: MIG is even larger.
Revised: MIG is even larger.

Original: Given the good performance of other methods on Shapes3D, we need more knowledge about our method’s relative failure on this benchmark.
Revised: Given the good performance of other methods on Shapes3D, we need more knowledge about our method’s relative failure on this benchmark.

Original: [Table]
Revised: [Table]

Original: To conclude, on the benchmarks and under the evaluation protocol provided by Locatello et al.
Revised: To conclude, on the benchmarks and under the evaluation protocol provided by Locatello et al.

Original: [Table]
Revised: [Table]

Original: our implemented contrastive learning shows SoTA-level performance in most cases.
Revised: our implemented contrastive learning shows SoTA-level performance in most cases.

Original: Its downsides on MIG and SAP metrics are probably related to its relaxed version of disentanglement property.
Revised: Its downsides on MIG and SAP metrics are probably related to its relaxed version of disentanglement property.

Original: The standard benchmark results help to provide a more accurate sense of the disentangling representation ability of contrastive learning.
Revised: The standard benchmark results help to provide a more accurate sense of the disentangling representation ability of contrastive learning.

Original: C M ORE Q UALITATIVE S TUDY
Revised: C M ORE Q UALITATIVE S TUDY

Original: Limited by the main text length limitation, we make a more qualitative study about the disentanglement property shown by the contrastive learning here.
Revised: Limited by the main text length limitation, we make a more qualitative study about the disentanglement property shown by the contrastive learning here.

Original: In Figure 4, we show the co-occurrence of mutual information of factors on dSprites by our model.
Revised: In Figure 4, we show the co-occurrence of mutual information of factors on dSprites by our model.

Original: We perform some qualitative studies on SmallNORB, Cars3D, and Shapes3D as well.
Revised: We perform some qualitative studies on SmallNORB, Cars3D, and Shapes3D as well.

Original: The visualization results are shown in Figure 6,
Revised: The visualization results are shown in Figure 6,

Original: Figure 8 and Figure 10.
Revised: Figure 8 and Figure 10.

Original: We can have some observation from them.
Revised: We can have some observation from them.

Original: SmallNORB For the results on SmallNORB in Figure 6, though most non-diagonal entries have very low co-occurrence of mutual information, two pairs of factors show slightly higher cooccurrence.
Revised: SmallNORB For the results on SmallNORB in Figure 6, though most non-diagonal entries have very low co-occurrence of mutual information, two pairs of factors show slightly higher cooccurrence.

Original: They are “azimuth-elevation” and “object/instance category-lighting”.
Revised: They are “azimuth-elevation” and “object/instance category-lighting”.

Original: After investigating the dataset, we ﬁnd the two pairs of factors are not fully independent.
Revised: After investigating the dataset, we ﬁnd the two pairs of factors are not fully independent.

Original: Figure 7 show some samples with corresponding factors manipulated.
Revised: Figure 7 show some samples with corresponding factors manipulated.

Original: We could see that the elevation and azimuth are not fully independent.
Revised: We could see that the elevation and azimuth are not fully independent.

Original: And the relation between the instance/object category and the lighting factor is even more obvious because the lighting condition is sensible by the shadow on and under the
Revised: And the relation between the instance/object category and the lighting factor is even more obvious because the lighting condition is sensible by the shadow on and under the

Original: [Table]
Revised: [Table]

Original: is reported in its original paper (Klindt et al., 2020).
Revised: is reported in its original paper (Klindt et al., 2020).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: instance, whose distribution and shape is highly determined by the instance category.
Revised: instance, whose distribution and shape is highly determined by the instance category.

Original: This reveals a common observation that, in real-world data (SmallNORB is not real-world yet
Revised: This reveals a common observation that, in real-world data (SmallNORB is not real-world yet

Original: but it is complicated than 2D synthetic images), the factors are usually hard to deﬁne to be fully independent or disentangled.
Revised: but it is complicated than 2D synthetic images), the factors are usually hard to deﬁne to be fully independent or disentangled.

Original: Cars3D For the results from Cars3D shown in Figure 8, only one pair of factors show some cooccurrence, i.e. “elevation-object type”.
Revised: Cars3D For the results from Cars3D shown in Figure 8, only one pair of factors show some cooccurrence, i.e. “elevation-object type”.

Original: To study how this happen, we randomly sampled some data from Cars3D by different object types and elevations as shown in Figure 9.
Revised: To study how this happen, we randomly sampled some data from Cars3D by different object types and elevations as shown in Figure 9.

Original: It shows that with the same value of elevation, samples of different object types help different visual elevation.
Revised: It shows that with the same value of elevation, samples of different object types help different visual elevation.

Original: So these two factors are not well disentangled.
Revised: So these two factors are not well disentangled.

Original: This might explain the slightly higher co-occurrence of mutual information between this pair of factors.
Revised: This might explain the slightly higher co-occurrence of mutual information between this pair of factors.

Original: Shapes3D The results on Shapes3D are shown in Figure 10.
Revised: Shapes3D The results on Shapes3D are shown in Figure 10.

Original: The result shows relatively bad disentanglement.
Revised: The result shows relatively bad disentanglement.

Original: To be precise, some factor pairs show low mutual information co-occurrence as expected, such as the color factors of ﬂoor, wall, and object and the pair of “object color - azimuth” .
Revised: To be precise, some factor pairs show low mutual information co-occurrence as expected, such as the color factors of ﬂoor, wall, and object and the pair of “object color - azimuth” .

Original: But the MI co-occurrence of “wall color - object size” and “object color - object size;” are higher than we expected as we did not recognize their high dependence.
Revised: But the MI co-occurrence of “wall color - object size” and “object color - object size;” are higher than we expected as we did not recognize their high dependence.

Original: This result might relate to our model’s relatively poor performance on Shapes3D as well.
Revised: This result might relate to our model’s relatively poor performance on Shapes3D as well.

Original: We hope the additional results shown above are helpful to provide more sense of the dataset conﬁguration and what we should expect from a well disentangled representation model.
Revised: We hope the additional results shown above are helpful to provide more sense of the dataset conﬁguration and what we should expect from a well disentangled representation model.

Original: [Figure]
Revised: [Figure]

Original: D M ORE A BLATION E XPERIMENTS
Revised: D M ORE A BLATION E XPERIMENTS

Original: Limited by the main content page length, we put some additional ablation study here to help better understand the inﬂuence of important inductive bias when studying representation disentanglement.
Revised: Limited by the main content page length, we put some additional ablation study here to help better understand the inﬂuence of important inductive bias when studying representation disentanglement.

Original: In the original implementation of contrastive learning
Revised: In the original implementation of contrastive learning

Original: [Table]
Revised: [Table]

Original: the min-ratio of Random-Resize-Crop is usually very small, e.g, 0.08 for BYOL or 0.2 for MoCo/SimCLR.
Revised: the min-ratio of Random-Resize-Crop is usually very small, e.g, 0.08 for BYOL or 0.2 for MoCo/SimCLR.

Original: However, we ﬁnd this hyperparameter is critical.
Revised: However, we ﬁnd this hyperparameter is critical.

Original: We thus perform a study with different minimum ratios of random-resize-crop
Revised: We thus perform a study with different minimum ratios of random-resize-crop

Original: [Table]
Revised: [Table]

Original: The results are shown in Table 10.
Revised: The results are shown in Table 10.

Original: From the table, we ﬁnd the disentanglement score ﬁrst increases when increasing the min-ratio of random-resize-crop but then drops.
Revised: From the table, we ﬁnd the disentanglement score ﬁrst increases when increasing the min-ratio of random-resize-crop but then drops.

Original: The default min-ratio is set to be 0.6 in our implementation.
Revised: The default min-ratio is set to be 0.6 in our implementation.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Learning rate is also another important hyper-parameter, we use Adam optimizer with learning rate to be 3 e − 4 as default.
Revised: Learning rate is also another important hyper-parameter, we use Adam optimizer with learning rate to be 3 e − 4 as default.

Original: But we are still interested in the inﬂuence of learning rate to the disentanglement property of learned representation.
Revised: But we are still interested in the inﬂuence of learning rate to the disentanglement property of learned representation.

Original: At the same time, batch normalization in the encoder network is another critical component.
Revised: At the same time, batch normalization in the encoder network is another critical component.

Original: In this part, we aim to study the transferability of learned representation, so we train BYOL under with different learning rates and batch normalization on three datasets respectively.
Revised: In this part, we aim to study the transferability of learned representation, so we train BYOL under with different learning rates and batch normalization on three datasets respectively.

Original: But the evaluation is always performed on the test set of dSprites.
Revised: But the evaluation is always performed on the test set of dSprites.

Original: The results are included in Table 12.
Revised: The results are included in Table 12.

Original: It shows, no norm not just shows a higher disentanglement score than BN but is also more robust with the variance of learning rate.
Revised: It shows, no norm not just shows a higher disentanglement score than BN but is also more robust with the variance of learning rate.

Original: When the learning rate is high, e.g., 3e-3 or 3e-2, the model with BN has collapsed already while the model without norm can still output a reasonably high disentanglement score.
Revised: When the learning rate is high, e.g., 3e-3 or 3e-2, the model with BN has collapsed already while the model without norm can still output a reasonably high disentanglement score.

Original: On the other hand, on the transferability from one dataset to another, the disentanglement score also shows high agreement.
Revised: On the other hand, on the transferability from one dataset to another, the disentanglement score also shows high agreement.

Original: [Table]
Revised: [Table]

Original: As the supplement of study on the normalization strategy, we provide a more detailed study in
Revised: As the supplement of study on the normalization strategy, we provide a more detailed study in

Original: Table 13 where ﬁve normalization strategies are evaluated with different batch sizes, which was found crucial in choosing normalization strategy in visual tasks.
Revised: Table 13 where ﬁve normalization strategies are evaluated with different batch sizes, which was found crucial in choosing normalization strategy in visual tasks.

Original: The result of the table still suggests that Batch Normalization weakens BYOL’s disentanglement property and no normalization, group normalization, and layer normalization show good disentanglement with different batch sizes for training.
Revised: The result of the table still suggests that Batch Normalization weakens BYOL’s disentanglement property and no normalization, group normalization, and layer normalization show good disentanglement with different batch sizes for training.

Original: When using instance normalization in the representation network, BYOL goes to collapse on dSprites.
Revised: When using instance normalization in the representation network, BYOL goes to collapse on dSprites.

Original: [Table]
Revised: [Table]

Original: Richemond et al.
Revised: Richemond et al.

Original: (2020) shows that it is possible to achieve similar results when replacing BN with group norm and weight standardization (Qiao et al., 2019).
Revised: (2020) shows that it is possible to achieve similar results when replacing BN with group norm and weight standardization (Qiao et al., 2019).

Original: We continue to show the inﬂuence of involving Group Normalization (GN) and weight standardization (WS) in Table 14.
Revised: We continue to show the inﬂuence of involving Group Normalization (GN) and weight standardization (WS) in Table 14.

Original: Here, we keep setting the group number to 4 and batch size to 512.
Revised: Here, we keep setting the group number to 4 and batch size to 512.

Original: And we use the two weight decay values in the original BYOL and Richemond et al.
Revised: And we use the two weight decay values in the original BYOL and Richemond et al.

Original: (2020) respectively (3e-8 and 1.5e-6).
Revised: (2020) respectively (3e-8 and 1.5e-6).

Original: The results show that compared with solely using GN, adding additional Weight Standardization can not just help learn features of higher quality but also be more disentangled.
Revised: The results show that compared with solely using GN, adding additional Weight Standardization can not just help learn features of higher quality but also be more disentangled.

Original: [Table]
Revised: [Table]