Original: R ETHINKING C ONTENT AND S TYLE : E XPLORING B IAS FOR U NSUPERVISED D ISENTANGLEMENT
Revised: R ETHINKING C ONTENT AND S TYLE : E XPLORING B IAS FOR U NSUPERVISED D ISENTANGLEMENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Content and style (C-S) disentanglement intends to decompose the underlying explanatory factors of objects into two independent latent spaces.
Revised: Content and style (C-S) disentanglement intends to decompose the underlying explanatory factors of objects into two independent latent spaces.

Original: Aiming for unsupervised disentanglement, we introduce an inductive bias to our formulation by assigning different and independent roles to content and style when approximating the real data distributions.
Revised: Aiming for unsupervised disentanglement, we introduce an inductive bias to our formulation by assigning different and independent roles to content and style when approximating the real data distributions.

Original: The content embeddings of individual images are forced to share a common distribution.
Revised: The content embeddings of individual images are forced to share a common distribution.

Original: The style embeddings encoding instancespeciﬁc features are used to customize the shared distribution.
Revised: The style embeddings encoding instancespeciﬁc features are used to customize the shared distribution.

Original: The experiments on several popular datasets demonstrate that our method achieves the state-of-theart disentanglement compared to other unsupervised approaches and comparable or even better results than supervised methods.
Revised: The experiments on several popular datasets demonstrate that our method achieves the state-of-theart disentanglement compared to other unsupervised approaches and comparable or even better results than supervised methods.

Original: Furthermore, as a new application of C-S disentanglement, we propose to generate multi-view images from a single view image for 3D reconstruction.
Revised: Furthermore, as a new application of C-S disentanglement, we propose to generate multi-view images from a single view image for 3D reconstruction.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The disentanglement task aims to recover the underlying explanatory factors of natural images into different dimensions of latent space, and provide an informative representation for tasks like image translation (Wu et al., 2019b; Kotovenko et al., 2019), domain adaptation (Li et al., 2019;
Revised: The disentanglement task aims to recover the underlying explanatory factors of natural images into different dimensions of latent space, and provide an informative representation for tasks like image translation (Wu et al., 2019b; Kotovenko et al., 2019), domain adaptation (Li et al., 2019;

Original: Zou et al., 2020) and geometric attributes extraction (Wu et al., 2019c;
Revised: Zou et al., 2020) and geometric attributes extraction (Wu et al., 2019c;

Original: Xing et al., 2019), etc.
Revised: Xing et al., 2019), etc.

Original: The previous methods (Kim & Mnih, 2018; Higgins et al., 2017; Burgess et al., 2018; Kumar et al., 2017) learn disentangled factors by optimizing the total correlation in an unsupervised manner.
Revised: The previous methods (Kim & Mnih, 2018; Higgins et al., 2017; Burgess et al., 2018; Kumar et al., 2017) learn disentangled factors by optimizing the total correlation in an unsupervised manner.

Original: However, Locatello et al.
Revised: However, Locatello et al.

Original: (2019) prove that unsupervised disentanglement is fundamentally impossible without inductive bias on both model and data.
Revised: (2019) prove that unsupervised disentanglement is fundamentally impossible without inductive bias on both model and data.

Original: In this paper, we focus on content and style (C-S) disentanglement, where content and style represent two separate groups of factors.
Revised: In this paper, we focus on content and style (C-S) disentanglement, where content and style represent two separate groups of factors.

Original: The main novelty of our work is that we assign different roles to the content and style in modeling the image distribution instead of treating the factors equally, which is the inductive bias introduced in our method.
Revised: The main novelty of our work is that we assign different roles to the content and style in modeling the image distribution instead of treating the factors equally, which is the inductive bias introduced in our method.

Original: Most of the previous C-S disentanglement works (Denton & Birodkar, 2017; Jha et al., 2018; Bouchacourt et al., 2018; Gabbay & Hoshen, 2020) rely on supervision, which is hard to obtain for real data.
Revised: Most of the previous C-S disentanglement works (Denton & Birodkar, 2017; Jha et al., 2018; Bouchacourt et al., 2018; Gabbay & Hoshen, 2020) rely on supervision, which is hard to obtain for real data.

Original: E.g., Gabbay & Hoshen (2020) leverage group observation to achieve disentanglement by forcing images from the same group to share a common embedding.
Revised: E.g., Gabbay & Hoshen (2020) leverage group observation to achieve disentanglement by forcing images from the same group to share a common embedding.

Original: To our best knowledge, the only exception is Wu et al. (2019c).
Revised: To our best knowledge, the only exception is Wu et al. (2019c).

Original: However, this method forces the content path to learn geometric structure limited by 2D landmarks.
Revised: However, this method forces the content path to learn geometric structure limited by 2D landmarks.

Original: Our deﬁnition of content and style is similar to Gabbay & Hoshen (2020), where the content includes the information which can be transferred among groups and style is image-speciﬁc information.
Revised: Our deﬁnition of content and style is similar to Gabbay & Hoshen (2020), where the content includes the information which can be transferred among groups and style is image-speciﬁc information.

Original: When group observation is not available, we deﬁne content includes the factors shared across the whole dataset, such as pose.
Revised: When group observation is not available, we deﬁne content includes the factors shared across the whole dataset, such as pose.

Original: Take the human face dataset CelebA (Liu et al., 2015) as an example, the content encodes pose, and style encodes identity, and multi-views of the same identity have the same style embeddings, but different content embeddings, i.e., poses.
Revised: Take the human face dataset CelebA (Liu et al., 2015) as an example, the content encodes pose, and style encodes identity, and multi-views of the same identity have the same style embeddings, but different content embeddings, i.e., poses.

Original: Based on the above deﬁnitions, we propose a new problem formulation and network architecture by introducing an inductive bias: assigning different and independent roles to content and style when approximating the real data distributions.
Revised: Based on the above deﬁnitions, we propose a new problem formulation and network architecture by introducing an inductive bias: assigning different and independent roles to content and style when approximating the real data distributions.

Original: Speciﬁcally, as shown in Figure 1, we force the content embeddings of individual images to share a common distribution, and the style embeddings are used to scale and shift the common distribution to match target image distribution via a generator.
Revised: Speciﬁcally, as shown in Figure 1, we force the content embeddings of individual images to share a common distribution, and the style embeddings are used to scale and shift the common distribution to match target image distribution via a generator.

Original: [Figure]
Revised: [Figure]

Original: Generator
Revised: Generator

Original: [Figure]
Revised: [Figure]

Original: 𝑠 𝑗 𝑠 𝑘 𝑐 𝑗 𝑐 𝑘 𝑠 𝑖 𝑐 𝑖 𝑓 𝜇 𝑓 𝜎 ×
Revised: 𝑠 𝑗 𝑠 𝑘 𝑐 𝑗 𝑐 𝑘 𝑠 𝑖 𝑐 𝑖 𝑓 𝜇 𝑓 𝜎 ×

Original: + 𝑠 𝑗 𝑠
Revised: + 𝑠 𝑗 𝑠

Original: 𝑘 𝑠 𝑖 𝑐 𝑗 𝑐 𝑘 𝑐 𝑖
Revised: 𝑘 𝑠 𝑖 𝑐 𝑗 𝑐 𝑘 𝑐 𝑖

Original: Sampling
Revised: Sampling

Original: C-S Fusion Block
Revised: C-S Fusion Block

Original: We follow Bojanowski et al.
Revised: We follow Bojanowski et al.

Original: and Gabbay & Hoshen (2020) to apply latent optimization to optimize the embeddings and the parameters of the generator.
Revised: and Gabbay & Hoshen (2020) to apply latent optimization to optimize the embeddings and the parameters of the generator.

Original: We also propose to use instance discrimination as a complementary constraint to assist the disentanglement.
Revised: We also propose to use instance discrimination as a complementary constraint to assist the disentanglement.

Original: Please note that we only use the image reconstruction loss as the supervision; no extra labeling is needed.
Revised: Please note that we only use the image reconstruction loss as the supervision; no extra labeling is needed.

Original: As the content and style perform a different and independent role when modeling the data, they are disentangled to encode the shared and instance-speciﬁc features respectively after the optimization.
Revised: As the content and style perform a different and independent role when modeling the data, they are disentangled to encode the shared and instance-speciﬁc features respectively after the optimization.

Original: The contributions of our work are as follows: we achieve unsupervised C-S disentanglement by introducing an inductive bias in our formulation: assign different and independent roles to content and style when modeling the real data distributions.
Revised: The contributions of our work are as follows: we achieve unsupervised C-S disentanglement by introducing an inductive bias in our formulation: assign different and independent roles to content and style when modeling the real data distributions.

Original: Furthermore, we achieve better C-S disentanglement by leveraging instance discrimination.
Revised: Furthermore, we achieve better C-S disentanglement by leveraging instance discrimination.

Original: The experiments on several popular datasets demonstrate that our method achieves the state-of-the-art unsupervised C-S disentanglement and comparable or even better results than supervised methods.
Revised: The experiments on several popular datasets demonstrate that our method achieves the state-of-the-art unsupervised C-S disentanglement and comparable or even better results than supervised methods.

Original: Besides, we propose to apply C-S disengagement to a new task: single view 3D reconstruction.
Revised: Besides, we propose to apply C-S disengagement to a new task: single view 3D reconstruction.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Unsupervised Disentanglement.
Revised: Unsupervised Disentanglement.

Original: A disentangled representation can be deﬁned as one where individual latent units are sensitive to changes in individual generative factors.
Revised: A disentangled representation can be deﬁned as one where individual latent units are sensitive to changes in individual generative factors.

Original: There have been a lot of studies on unsupervised disentangled representation learning (Higgins et al., 2017; Burgess et al., 2018; Kumar et al., 2017; Kim & Mnih, 2018; Chen et al., 2018).
Revised: There have been a lot of studies on unsupervised disentangled representation learning (Higgins et al., 2017; Burgess et al., 2018; Kumar et al., 2017; Kim & Mnih, 2018; Chen et al., 2018).

Original: These models learn disentangled factors by factorizing aggregated posterior.
Revised: These models learn disentangled factors by factorizing aggregated posterior.

Original: They can also be used for C-S disentanglement.
Revised: They can also be used for C-S disentanglement.

Original: The learned factors can be divided into two categories; one is content-related, the other is style-related.
Revised: The learned factors can be divided into two categories; one is content-related, the other is style-related.

Original: However, Locatello et al.
Revised: However, Locatello et al.

Original: (2019) proved that unsupervised disentanglement is impossible without introducing inductive bias on both models and data.
Revised: (2019) proved that unsupervised disentanglement is impossible without introducing inductive bias on both models and data.

Original: Therefore, these models are currently unable to obtain a promising disentangled representation.
Revised: Therefore, these models are currently unable to obtain a promising disentangled representation.

Original: Motivated by Locatello et al.
Revised: Motivated by Locatello et al.

Original: (2019), we revisit and formulate the unsupervised C-S disentanglement problem to introduce inductive bias.
Revised: (2019), we revisit and formulate the unsupervised C-S disentanglement problem to introduce inductive bias.

Original: C-S Disentanglement.
Revised: C-S Disentanglement.

Original: Originated from style transfer, most of the prior works on C-S disentanglement divide latent variables into two spaces relying on supervision.
Revised: Originated from style transfer, most of the prior works on C-S disentanglement divide latent variables into two spaces relying on supervision.

Original: To achieve disentanglement,
Revised: To achieve disentanglement,

Original: Mathieu et al.
Revised: Mathieu et al.

Original: (2016) and Szab´o et al.
Revised: (2016) and Szab´o et al.

Original: (2018) combine the adversarial constraint and auto-encoders.
Revised: (2018) combine the adversarial constraint and auto-encoders.

Original: Meanwhile, VAE (Kingma & Welling, 2014) is used with non-adversarial constraints, such as cycle consistency (Jha et al., 2018) and evidence accumulation (Bouchacourt et al., 2018).
Revised: Meanwhile, VAE (Kingma & Welling, 2014) is used with non-adversarial constraints, such as cycle consistency (Jha et al., 2018) and evidence accumulation (Bouchacourt et al., 2018).

Original: Furthermore, latent optimization is shown to be superior to amortized inference (Gabbay & Hoshen, 2020).
Revised: Furthermore, latent optimization is shown to be superior to amortized inference (Gabbay & Hoshen, 2020).

Original: Unlike the above works, Wu et al. (2019c) propose a variational U-Net with structure learning for disentanglement in an unsupervised manner.
Revised: Unlike the above works, Wu et al. (2019c) propose a variational U-Net with structure learning for disentanglement in an unsupervised manner.

Original: However, this method is limited by the learning of 2D landmarks.
Revised: However, this method is limited by the learning of 2D landmarks.

Original: In our paper, we formulate C-S disentanglement and explore inductive bias for unsupervised disentanglement.
Revised: In our paper, we formulate C-S disentanglement and explore inductive bias for unsupervised disentanglement.

Original: Note that style transfer aims at modifying the domain style of an image while preserving its content, and its formulation focuses on the relation between domains (Huang et al., 2018a).
Revised: Note that style transfer aims at modifying the domain style of an image while preserving its content, and its formulation focuses on the relation between domains (Huang et al., 2018a).

Original: Our formulation is deﬁned in a single domain but can be extended to cross-domain, as presented in Appendix
Revised: Our formulation is deﬁned in a single domain but can be extended to cross-domain, as presented in Appendix

Original: I
Revised: I

Original:  E XPLORING I NDUCTIVE B IAS FOR C-S D ISENTANGLEMENT
Revised:  E XPLORING I NDUCTIVE B IAS FOR C-S D ISENTANGLEMENT

Original: In this section, We ﬁrst formulate the C-S disentangle problem by exploring the inductive bias and propose a C-S fusion block based on our formulation.
Revised: In this section, We ﬁrst formulate the C-S disentangle problem by exploring the inductive bias and propose a C-S fusion block based on our formulation.

Original: We then perform the ablation study to demonstrate how the C-S get disentangled.
Revised: We then perform the ablation study to demonstrate how the C-S get disentangled.

Original: Finally, our loss functions are presented.
Revised: Finally, our loss functions are presented.

Original: 3. P ROBLEM F ORMULATION
Revised: 3. P ROBLEM F ORMULATION

Original: We parametric the target distribution P i ( x | c ) as ˆ P θ,s i ( x | c ) , where θ is the parameter of the generator
Revised: We parametric the target distribution P i ( x | c ) as ˆ P θ,s i ( x | c ) , where θ is the parameter of the generator

Original: G θ that maps embeddings to images, s i is the style embedding assigned to I i .
Revised: G θ that maps embeddings to images, s i is the style embedding assigned to I i .

Original: In our formulation of ˆ P , we assign independent roles to content and style embeddings.
Revised: In our formulation of ˆ P , we assign independent roles to content and style embeddings.

Original: { c i } Ni =1 are sampled from the distribution of the shared conditional variable c , which is denoted as Ψ .
Revised: { c i } Ni =1 are sampled from the distribution of the shared conditional variable c , which is denoted as Ψ .

Original: { s i } Ni =1 are the parameters to characterize ˆ P .
Revised: { s i } Ni =1 are the parameters to characterize ˆ P .

Original: Thus our inductive bias is introduced into our formulation.
Revised: Thus our inductive bias is introduced into our formulation.

Original: Ψ should be close to the ground truth distribution of the dataset, e.g., Gaussian distribution, Uniform distribution, etc.
Revised: Ψ should be close to the ground truth distribution of the dataset, e.g., Gaussian distribution, Uniform distribution, etc.

Original: Our optimization target is to maximize the log-likelihood of ˆ P , and force c to follow the shared distribution Ψ meanwhile:
Revised: Our optimization target is to maximize the log-likelihood of ˆ P , and force c to follow the shared distribution Ψ meanwhile:

Original: [Equation]
Revised: [Equation]

Original: where c = c i indicates c
Revised: where c = c i indicates c

Original: i is the embedding assigned to I i , and p ( c ) denotes the distribution of { c i } Ni =1 .
Revised: i is the embedding assigned to I i , and p ( c ) denotes the distribution of { c i } Ni =1 .

Original: To solve this problem, we introduce the Lagrange Multiplier as
Revised: To solve this problem, we introduce the Lagrange Multiplier as

Original: [Equation]
Revised: [Equation]

Original: 3. P ROPOSED N ETWORK A RCHITECTURE
Revised: 3. P ROPOSED N ETWORK A RCHITECTURE

Original: Here we propose a network architecture to address the formulated problem in Section.
Revised: Here we propose a network architecture to address the formulated problem in Section.

Original: In particular, we design a C-S fusion block to assign different roles to content and style in modeling real data distribution.
Revised: In particular, we design a C-S fusion block to assign different roles to content and style in modeling real data distribution.

Original: As shown in Figure 1, we add this block before the generator to force the input to follow the customized distribution.
Revised: As shown in Figure 1, we add this block before the generator to force the input to follow the customized distribution.

Original: Inspired by the observation that mean and variance of features carry the style information (Gatys et al., 2016; Li & Wand, 2016; Li et al., 2017; Huang & Belongie, 2017), we use the style embeddings to provide the statistics to scale and shift the shared distribution Ψ to match the target distribution as
Revised: Inspired by the observation that mean and variance of features carry the style information (Gatys et al., 2016; Li & Wand, 2016; Li et al., 2017; Huang & Belongie, 2017), we use the style embeddings to provide the statistics to scale and shift the shared distribution Ψ to match the target distribution as

Original: [Equation]
Revised: [Equation]

Original: where f σ and f µ are two fully connected layers predicting the mean and variance respectively.
Revised: where f σ and f µ are two fully connected layers predicting the mean and variance respectively.

Original: With this design, Eq. 1 is equivalent to minimize
Revised: With this design, Eq. 1 is equivalent to minimize

Original: [Equation]
Revised: [Equation]

Original: Please refer to Appendix J for the proof.
Revised: Please refer to Appendix I for the proof.

Original: To solve Eq. 4, for the reconstruction term, we adopt latent optimization to optimize θ, { c i } Ni =1 , { s i } Ni =1 .
Revised: To solve Eq. 4, for the reconstruction term, we adopt latent optimization to optimize θ, { c i } Ni =1 , { s i } Ni =1 .

Original: For the KL term, it can not be optimised directly.
Revised: For the KL term, it can not be optimised directly.

Original: However, when Ψ has some speciﬁc forms, we can adopt a normalization to force each of the content embeddings
Revised: However, when Ψ has some speciﬁc forms, we can adopt a normalization to force each of the content embeddings

Original: c i to follow it approximately.
Revised: c i to follow it approximately.

Original: We can choose different forms of Ψ to ﬁt the ground truth distribution when solving the optimization problem, here we provide two examples:
Revised: We can choose different forms of Ψ to ﬁt the ground truth distribution when solving the optimization problem, here we provide two examples:

Original: Gaussian Distribution.
Revised: Gaussian Distribution.

Original: Gaussian distribution is used to model the distribution of images in many works (Kingma & Welling, 2014;
Revised: Gaussian distribution is used to model the distribution of images in many works (Kingma & Welling, 2014;

Original: Kim & Mnih, 2018; Higgins et al., 2017).
Revised: Kim & Mnih, 2018; Higgins et al., 2017).

Original: When setting the shared distribution Ψ as a zero mean, unit variance Gaussian distribution N (0 , I ) , we can use instance normalization (IN) to force each of the content embeddings
Revised: When setting the shared distribution Ψ as a zero mean, unit variance Gaussian distribution N (0 , I ) , we can use instance normalization (IN) to force each of the content embeddings

Original: c i to follow N (0 , I ) in the optimization
Revised: c i to follow N (0 , I ) in the optimization

Original: [Figure]
Revised: [Figure]

Original: process.
Revised: process.

Original: By combining IN and Eq. 3, we get the same formulation as AdaIN (Huang & Belongie, 2017), which is widely adopted in style transfer tasks (Huang et al., 2018a; Kotovenko et al., 2019).
Revised: By combining IN and Eq. 3, we get the same formulation as AdaIN (Huang & Belongie, 2017), which is widely adopted in style transfer tasks (Huang et al., 2018a; Kotovenko et al., 2019).

Original: Normalizing the feature map of the network to Gaussian is helpful for network training (Ioffe &
Revised: Normalizing the feature map of the network to Gaussian is helpful for network training (Ioffe &

Original: Szegedy, 2015; Wu & He, 2018), but our motivation for using normalization is to force the embeddings to share the same Gaussian distribution, which differs from these works.
Revised: Szegedy, 2015; Wu & He, 2018), but our motivation for using normalization is to force the embeddings to share the same Gaussian distribution, which differs from these works.

Original: Uniform Distribution.
Revised: Uniform Distribution.

Original: In many datasets, the distribution of content is close to a uniform distribution, e.g., in the Chairs (Aubry et al., 2014) dataset, the images are synthesized from dense views surrounding the objects.
Revised: In many datasets, the distribution of content is close to a uniform distribution, e.g., in the Chairs (Aubry et al., 2014) dataset, the images are synthesized from dense views surrounding the objects.

Original: For these datasets, we set Ψ to be uniform distribution, and normalize the content embeddings with L 2 normalization to force each of them to follow uniform distribution approximately (Muller, 1959).
Revised: For these datasets, we set Ψ to be uniform distribution, and normalize the content embeddings with L 2 normalization to force each of them to follow uniform distribution approximately (Muller, 1959).

Original: As shown in Figure 1, we can use this C-S fusion block only before the generator, denoted as the
Revised: As shown in Figure 1, we can use this C-S fusion block only before the generator, denoted as the

Original: Single C-S Fusion framework.
Revised: Single C-S Fusion framework.

Original: We can also provide multiple paths to implement our design by inserting it before every layer of the generator, denoted as a Multiple C-S Fusion framework.
Revised: We can also provide multiple paths to implement our design by inserting it before every layer of the generator, denoted as a Multiple C-S Fusion framework.

Original: For details of the network structures, please refer to Appendix A.2.
Revised: For details of the network structures, please refer to Appendix A.2.

Original: 3. D EMYSTIFYING C-S DISENTANGLEMENT
Revised: 3. D EMYSTIFYING C-S DISENTANGLEMENT

Original: In this subsection, we perform some experiments to verify that assigning different and independent roles to content and style for modeling real data distribution is the key to C-S disentanglement.
Revised: In this subsection, we perform some experiments to verify that assigning different and independent roles to content and style for modeling real data distribution is the key to C-S disentanglement.

Original: The experimental setting can be found in Section 4.
Revised: The experimental setting can be found in Section 4.

Original: If we do not assign different roles, i.e., concatenating content and style embedding as the input of the generator, the network can hardly disentangle any meaningful information for the CelebA dataset, as shown in Figure 2 (a).
Revised: If we do not assign different roles, i.e., concatenating content and style embedding as the input of the generator, the network can hardly disentangle any meaningful information for the CelebA dataset, as shown in Figure 2 (a).

Original: Our Single C-S Fusion framework can disentangle the pose and identity of human faces, as shown in Figure 2 (c).
Revised: Our Single C-S Fusion framework can disentangle the pose and identity of human faces, as shown in Figure 2 (c).

Original: The content plays the role of modeling the shared distribution.
Revised: The content plays the role of modeling the shared distribution.

Original: When the shared distribution constraint is removed, i.e., without normalization, the result is shown in Figure 2 (b), where the pose and identity can not be disentangled.
Revised: When the shared distribution constraint is removed, i.e., without normalization, the result is shown in Figure 2 (b), where the pose and identity can not be disentangled.

Original: For the Multiple C-S Fusion framework, multiple paths are provided, and the network has more ﬂexibility to approximate the target distribution and outperforms the Single C-S Fusion framework, as shown in Figure 2 (d).
Revised: For the Multiple C-S Fusion framework, multiple paths are provided, and the network has more ﬂexibility to approximate the target distribution and outperforms the Single C-S Fusion framework, as shown in Figure 2 (d).

Original: Since the shared distribution is crucial, we experiment to demonstrate that better disentanglement can be achieved by choosing a better distribution to ﬁt the dataset.
Revised: Since the shared distribution is crucial, we experiment to demonstrate that better disentanglement can be achieved by choosing a better distribution to ﬁt the dataset.

Original: For the real-world dataset CelebA, the distribution of pose is better modeled as a Gaussian distribution.
Revised: For the real-world dataset CelebA, the distribution of pose is better modeled as a Gaussian distribution.

Original: As Figure 3 (a) and (b) show,
Revised: As Figure 3 (a) and (b) show,

Original: IN achieves better disentanglement than L 2 .
Revised: IN achieves better disentanglement than L 2 .

Original: For the synthetic Chairs (Aubry et al., 2014) dataset, the distribution of pose is close to uniform distribution rather than Gaussian distribution.
Revised: For the synthetic Chairs (Aubry et al., 2014) dataset, the distribution of pose is close to uniform distribution rather than Gaussian distribution.

Original: Figure(c) and (d) show that the L 2 normalization results in better identity and pose consistency.
Revised: Figure(c) and (d) show that the L 2 normalization results in better identity and pose consistency.

Original: We also provide the generated images during the training process in the Appendix F, showing that the shared content is disentangled ﬁrst and then the instance-speciﬁc style.
Revised: We also provide the generated images during the training process in the Appendix F, showing that the shared content is disentangled ﬁrst and then the instance-speciﬁc style.

Original: 3. L OSS F UNCTION
Revised: 3. L OSS F UNCTION

Original: Perceptual Loss.
Revised: Perceptual Loss.

Original: Perceptual Loss is widely used in weakly supervised and unsupervised methods (Wu et al., 2020; Ren et al., 2020; Wu et al., 2019c).
Revised: Perceptual Loss is widely used in weakly supervised and unsupervised methods (Wu et al., 2020; Ren et al., 2020; Wu et al., 2019c).

Original: Gabbay & Hoshen (2020) claimed that perceptual loss is not extra supervision for disentanglement.
Revised: Gabbay & Hoshen (2020) claimed that perceptual loss is not extra supervision for disentanglement.

Original: We adopt a VGG (Simonyan & Zisserman, 2015) perceptual loss L P as a reconstruction loss in Eq.
Revised: We adopt a VGG (Simonyan & Zisserman, 2015) perceptual loss L P as a reconstruction loss in Eq.

Original: 4, implemented by Hoshen et al.
Revised: 4, implemented by Hoshen et al.

Original: [Figure]
Revised: [Figure]

Original: Instance Discrimination.
Revised: Instance Discrimination.

Original: Instance discrimination can automatically discover appearance similarity among semantic categories (Wu et al., 2018).
Revised: Instance discrimination can automatically discover appearance similarity among semantic categories (Wu et al., 2018).

Original: Inspired by this, we propose to use instance discrimination as a complementary constraint to enhance consistency among the images sharing the same style embeddings.
Revised: Inspired by this, we propose to use instance discrimination as a complementary constraint to enhance consistency among the images sharing the same style embeddings.

Original: We denote the instance discrimination loss as L ID .
Revised: We denote the instance discrimination loss as L ID .

Original: The implementation detail can be found in Appendix C.3.
Revised: The implementation detail can be found in Appendix C.3.

Original: Information Bottleneck.
Revised: Information Bottleneck.

Original: Burgess et al.
Revised: Burgess et al.

Original: (2018) propose improving the disentanglement in β -VAE by controlling the capacity increment, i.e., forcing the KL divergence to be a controllable value.
Revised: (2018) propose improving the disentanglement in β -VAE by controlling the capacity increment, i.e., forcing the KL divergence to be a controllable value.

Original: This motivated us to control the information bottleneck capacity of content and style to help to avoid leakage.
Revised: This motivated us to control the information bottleneck capacity of content and style to help to avoid leakage.

Original: This loss is denoted as L IB .
Revised: This loss is denoted as L IB .

Original: The details of this loss are provided in Appendix C.4.
Revised: The details of this loss are provided in Appendix C.4.

Original: Our full objective is
Revised: Our full objective is

Original: [Equation]
Revised: [Equation]

Original: where hyperparameters w P , w IB , and w ID represent the weights for each loss term respectively.
Revised: where hyperparameters w P , w IB , and w ID represent the weights for each loss term respectively.

Original: The ablation study for the loss terms is presented in Appendix E.
Revised: The ablation study for the loss terms is presented in Appendix E.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we perform quantitative and qualitative experiments to evaluate our method on seen data following common practice.
Revised: In this section, we perform quantitative and qualitative experiments to evaluate our method on seen data following common practice.

Original: For unseen image inference, please refer to Appendix G. We test our method on several datasets:
Revised: For unseen image inference, please refer to Appendix G. We test our method on several datasets:

Original: Car3D (Reed et al., 2015), Chairs (Aubry et al., 2014), CelebA (Liu et al., 2015) and Market-1501 (Zheng et al., 2015).
Revised: Car3D (Reed et al., 2015), Chairs (Aubry et al., 2014), CelebA (Liu et al., 2015) and Market-1501 (Zheng et al., 2015).

Original: For details of the datasets, please refer to
Revised: For details of the datasets, please refer to

Original: Appendix B.
Revised: Appendix B.

Original: Baselines.
Revised: Baselines.

Original: Among all the prior works, we choose several state-of-the-art class-supervised C-S disentanglement benchmarks for comparisons: Cycle-VAE (Jha et al., 2018), a variant of VAE using cycle-consistency; DrNet (Denton & Birodkar, 2017), an adversarial approach; Lord (Gabbay &
Revised: Among all the prior works, we choose several state-of-the-art class-supervised C-S disentanglement benchmarks for comparisons: Cycle-VAE (Jha et al., 2018), a variant of VAE using cycle-consistency; DrNet (Denton & Birodkar, 2017), an adversarial approach; Lord (Gabbay &

Original: Hoshen, 2020), a latent optimization method.
Revised: Hoshen, 2020), a latent optimization method.

Original: We also choose two unsupervised disentanglement methods: FactorVAE (Kim & Mnih, 2018), a method that encourages the distribution of representations to be factorial; Wu et al.
Revised: We also choose two unsupervised disentanglement methods: FactorVAE (Kim & Mnih, 2018), a method that encourages the distribution of representations to be factorial; Wu et al.

Original: (2019c) 1 , a two-branch VAE framework based on unsupervised structure learning.
Revised: (2019c) 1 , a two-branch VAE framework based on unsupervised structure learning.

Original: More details for baselines are presented in Appendix B.
Revised: More details for baselines are presented in Appendix B.

Original: 4. Q UANTITATIVE E XPERIMENTS
Revised: 4. Q UANTITATIVE E XPERIMENTS

Original: We compare our method (Multiple C-S Fusion framework) with the baselines on Car3D, Chairs and
Revised: We compare our method (Multiple C-S Fusion framework) with the baselines on Car3D, Chairs and

Original: CelebA.
Revised: CelebA.

Original: Content Transfer Metric.
Revised: Content Transfer Metric.

Original: To evaluate our method’s disentanglement ability, we follow the protocol of Gabbay & Hoshen (2020) to measure the quality of content transfer by LPIPS (Zhang et al., 2018).
Revised: To evaluate our method’s disentanglement ability, we follow the protocol of Gabbay & Hoshen (2020) to measure the quality of content transfer by LPIPS (Zhang et al., 2018).

Original: Details are presented in Appendix A.1.
Revised: Details are presented in Appendix A.1.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: We achieve the best performance among the unsupervised methods, even though pseudo label is provided for Wu et al. (2019c).
Revised: We achieve the best performance among the unsupervised methods, even though pseudo label is provided for Wu et al. (2019c).

Original: Furthermore, our method is comparable to or even better than the supervised ones.
Revised: Furthermore, our method is comparable to or even better than the supervised ones.

Original: Classiﬁcation Metric.
Revised: Classiﬁcation Metric.

Original: Classiﬁcation accuracy is used to evaluate disentanglement in the literature (Denton & Birodkar, 2017; Jha et al., 2018; Gabbay & Hoshen, 2020).
Revised: Classiﬁcation accuracy is used to evaluate disentanglement in the literature (Denton & Birodkar, 2017; Jha et al., 2018; Gabbay & Hoshen, 2020).

Original: Following Jha et al.
Revised: Following Jha et al.

Original: (2018), we train two models of a single fully-connected layer to classify content labels from style embeddings and classify style labels from content embeddings.
Revised: (2018), we train two models of a single fully-connected layer to classify content labels from style embeddings and classify style labels from content embeddings.

Original: Low classiﬁcation accuracy indicates that
Revised: Low classiﬁcation accuracy indicates that

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: (c) (d) (e) (f) the leakage between content and style is small.
Revised: (c) (d) (e) (f) the leakage between content and style is small.

Original: Due to no content annotations of CelebA, we regress the position of the facial landmarks from the style embeddings.
Revised: Due to no content annotations of CelebA, we regress the position of the facial landmarks from the style embeddings.

Original: The results are summarized in Table 2.
Revised: The results are summarized in Table 2.

Original: Though without supervision, our method is comparable to several methods.
Revised: Though without supervision, our method is comparable to several methods.

Original: We observe that the classiﬁcation metric is also inﬂuenced by information capacity and dimensions of embeddings.
Revised: We observe that the classiﬁcation metric is also inﬂuenced by information capacity and dimensions of embeddings.

Original: For FactorVAE (Kim & Mnih, 2018), the poor reconstruction quality indicates that the latent embeddings encode a very small amount of information that can hardly be classiﬁed.
Revised: For FactorVAE (Kim & Mnih, 2018), the poor reconstruction quality indicates that the latent embeddings encode a very small amount of information that can hardly be classiﬁed.

Original: The dimensions of the latent vectors of different methods vary from ten to hundreds.
Revised: The dimensions of the latent vectors of different methods vary from ten to hundreds.

Original: Actually, the higher dimension usually leads to easier classiﬁcation.
Revised: Actually, the higher dimension usually leads to easier classiﬁcation.

Original: Based on the above observations, the classiﬁcation metric may not be appropriate for disentanglement, which is also observed in Liu et al.
Revised: Based on the above observations, the classiﬁcation metric may not be appropriate for disentanglement, which is also observed in Liu et al.

Original: 4. Q UALITATIVE E XPERIMENTS
Revised: 4. Q UALITATIVE E XPERIMENTS

Original: Disentanglement & alignment.
Revised: Disentanglement & alignment.

Original: In Figure 4 (a) and (b), we conduct linear interpolation to show the variation in the two latent manifolds.
Revised: In Figure 4 (a) and (b), we conduct linear interpolation to show the variation in the two latent manifolds.

Original: Horizontally, the identity is changed smoothly with the interpolated style latent space while maintaining the pose information.
Revised: Horizontally, the identity is changed smoothly with the interpolated style latent space while maintaining the pose information.

Original: Vertically, the identity remains the same as the pose changes.
Revised: Vertically, the identity remains the same as the pose changes.

Original: These results illustrate the following points: 1) The learned content and style spaces are continuous.
Revised: These results illustrate the following points: 1) The learned content and style spaces are continuous.

Original: 2) Columns of the left and right ﬁgures share the same pose, suggesting that the learned content spaces are aligned.
Revised: 2) Columns of the left and right ﬁgures share the same pose, suggesting that the learned content spaces are aligned.

Original: 3) Style-related information is maintained when changing the content embedding and vice versa, suggesting the good disentanglement.
Revised: 3) Style-related information is maintained when changing the content embedding and vice versa, suggesting the good disentanglement.

Original: We perform retrieval on the content and style latent spaces, respectively.
Revised: We perform retrieval on the content and style latent spaces, respectively.

Original: As shown in Figure 4 (c) and (d), the nearest neighbors in the content space share the same pose but have different identities, which reveals the alignment on content space.
Revised: As shown in Figure 4 (c) and (d), the nearest neighbors in the content space share the same pose but have different identities, which reveals the alignment on content space.

Original: To better identify the faces, we let the nearest neighbors in the style space share the same pose, and the generated faces look very similar, revealing that the style is well maintained.
Revised: To better identify the faces, we let the nearest neighbors in the style space share the same pose, and the generated faces look very similar, revealing that the style is well maintained.

Original: As shown in Figure 4 (f), one interesting observation is that zero content embeddings lead to a canonical view.
Revised: As shown in Figure 4 (f), one interesting observation is that zero content embeddings lead to a canonical view.

Original: As we assume that the pose distribution of faces is
Revised: As we assume that the pose distribution of faces is

Original: N (0 , I ) , the canonical view is the most common pose in the dataset and sampled from the peak of this distribution.
Revised: N (0 , I ) , the canonical view is the most common pose in the dataset and sampled from the peak of this distribution.

Original: We also show the faces with zero style embeddings in Figure 4 (e), and it looks like the mean face of the dataset.
Revised: We also show the faces with zero style embeddings in Figure 4 (e), and it looks like the mean face of the dataset.

Original: Visual Analogy & Comparison.
Revised: Visual Analogy & Comparison.

Original: Visual analogy (Reed et al., 2015) is to switch between style and content embeddings for each pair within a testset.
Revised: Visual analogy (Reed et al., 2015) is to switch between style and content embeddings for each pair within a testset.

Original: We show the visual analogy results of our method against FactorVAE (Kim & Mnih, 2018) (typical unsupervised baseline) and Lord (strongest super-
Revised: We show the visual analogy results of our method against FactorVAE (Kim & Mnih, 2018) (typical unsupervised baseline) and Lord (strongest super-

Original: [Table]
Revised: [Table]

Original: vised baseline) in Figure 6 on Chairs, Car3D, and CelebA. The results of FactorVAE on all datasets have poor reconstruction quality and bad content transfer.
Revised: vised baseline) in Figure 6 on Chairs, Car3D, and CelebA. The results of FactorVAE on all datasets have poor reconstruction quality and bad content transfer.

Original: On Cars3D, Lord has artifacts (e.g., third column) and could not capture the color style of the test images (e.g., fourth row).
Revised: On Cars3D, Lord has artifacts (e.g., third column) and could not capture the color style of the test images (e.g., fourth row).

Original: On CelebA, the transfer result of Lord has ambiguity, e.g., the content embedding controls facial expression in the ﬁfth column, while other content embeddings do not control expression.
Revised: On CelebA, the transfer result of Lord has ambiguity, e.g., the content embedding controls facial expression in the ﬁfth column, while other content embeddings do not control expression.

Original: Our method achieves comparable pose transfer to Lord and maintains the identity of the images.
Revised: Our method achieves comparable pose transfer to Lord and maintains the identity of the images.

Original: Furthermore, we show our results on the Market-1501 dataset in Figure 7, which demonstrates our method can disentangle the human pose and the appearance even though the skeletons have large variances.
Revised: Furthermore, we show our results on the Market-1501 dataset in Figure 7, which demonstrates our method can disentangle the human pose and the appearance even though the skeletons have large variances.

Original: For more results (including other datasets), please refer to Appendix D.
Revised: For more results (including other datasets), please refer to Appendix D.

Original:  N EW A PPLICATION
Revised:  N EW A PPLICATION

Original: In this work, we explore a new application of C-S disentanglement.
Revised: In this work, we explore a new application of C-S disentanglement.

Original: For 3D reconstruction, singleview settings lack reliable 3D constraints, which can cause unresolvable ambiguities (Wu et al., 2019a).
Revised: For 3D reconstruction, singleview settings lack reliable 3D constraints, which can cause unresolvable ambiguities (Wu et al., 2019a).

Original: Thanks to our disentangled representations, we can generate multi-view images from a single view by extracting the style embedding of the single view and then combining it with multiple content embeddings.
Revised: Thanks to our disentangled representations, we can generate multi-view images from a single view by extracting the style embedding of the single view and then combining it with multiple content embeddings.

Original: On Chairs, we adopt Pix2Vox (Xie et al., 2019), a framework for single-view, and multi-view 3D reconstruction to verify the advantages of our method.
Revised: On Chairs, we adopt Pix2Vox (Xie et al., 2019), a framework for single-view, and multi-view 3D reconstruction to verify the advantages of our method.

Original: As shown in Figure 5, the 3D objects reconstructed from multi-view inputs generated from our method are much better than those reconstructed from a single view, and even comparable to those reconstructed from groundtruth multi-view images.
Revised: As shown in Figure 5, the 3D objects reconstructed from multi-view inputs generated from our method are much better than those reconstructed from a single view, and even comparable to those reconstructed from groundtruth multi-view images.

Original: For results on Celeba, please refer to Appendix D.3.
Revised: For results on Celeba, please refer to Appendix D.3.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present an unsupervised C-S disentanglement method, based on an inductive bias: assigning different and independent roles to content and style when approximating the real data distributions.
Revised: We present an unsupervised C-S disentanglement method, based on an inductive bias: assigning different and independent roles to content and style when approximating the real data distributions.

Original: Our method outperforms other unsupervised approaches and achieves comparable to or even better performance than the state-of-the-art supervised methods.
Revised: Our method outperforms other unsupervised approaches and achieves comparable to or even better performance than the state-of-the-art supervised methods.

Original: We also propose to use it to help single-view 3D reconstruction, as a new application of C-S disentanglement.
Revised: We also propose to use it to help single-view 3D reconstruction, as a new application of C-S disentanglement.

Original: As for the limitation, we fail on
Revised: As for the limitation, we fail on

Original: [Figure]
Revised: [Figure]

Original: datasets containing multiple categories with large appearance variation, e.g., CIFAR-10 (Krizhevsky et al., 2009), which does not match our assumption.
Revised: datasets containing multiple categories with large appearance variation, e.g., CIFAR-10 (Krizhevsky et al., 2009), which does not match our assumption.

Original: Our method could be adopted to help downstream tasks, e.g., domain translation, ReID, etc.
Revised: Our method could be adopted to help downstream tasks, e.g., domain translation, ReID, etc.

Original: An interesting direction is to apply our method to instance discrimination.
Revised: An interesting direction is to apply our method to instance discrimination.

Original: With disentangled representations, contrastive learning is expected to perform more effectively.
Revised: With disentangled representations, contrastive learning is expected to perform more effectively.

Original: A I MPLEMENTATION D ETAILS
Revised: A I MPLEMENTATION D ETAILS

Original: All the models are trained on an Nvidia Tesla V100 GPU.
Revised: All the models are trained on an Nvidia Tesla V100 GPU.

Original: The model is implemented in PyTorch (Paszke et al., 2017).
Revised: The model is implemented in PyTorch (Paszke et al., 2017).

Original: For Car3D, Chairs, and Celeba, we set the size of the style embedding d s to 256, the size of the content embedding d c to 128, and the epoch to 200.
Revised: For Car3D, Chairs, and Celeba, we set the size of the style embedding d s to 256, the size of the content embedding d c to 128, and the epoch to 200.

Original: All the images are resized to 64 × 64 .
Revised: All the images are resized to 64 × 64 .

Original: For the hyperparameters, w P = 5 , w IB = 1 , w ID = 1 .
Revised: For the hyperparameters, w P = 5 , w IB = 1 , w ID = 1 .

Original: We used Adam (Kingma & Ba, 2015) with a learning rate of 0.0003 for the models and 0.003 for the latent spaces.
Revised: We used Adam (Kingma & Ba, 2015) with a learning rate of 0.0003 for the models and 0.003 for the latent spaces.

Original: A. E XPERIMENTAL D ETAILS
Revised: A. E XPERIMENTAL D ETAILS

Original: Content Transfer Metric.
Revised: Content Transfer Metric.

Original: For Car3D and Chairs datasets, the content labels (ground truth) are available.
Revised: For Car3D and Chairs datasets, the content labels (ground truth) are available.

Original: For images I i and I j sampled from the testing set randomly, we compute LPIPS between
Revised: For images I i and I j sampled from the testing set randomly, we compute LPIPS between

Original: G θ ( c i , s j ) and the corresponding ground truth from the same class of I j .
Revised: G θ ( c i , s j ) and the corresponding ground truth from the same class of I j .

Original: For CelebA, we randomly sample two images I i and I j with the same identity, and we retrieve an image that has the most similar pose k from the test set, i.e., the nearest neighbor in the 68 facial-landmarks space.
Revised: For CelebA, we randomly sample two images I i and I j with the same identity, and we retrieve an image that has the most similar pose k from the test set, i.e., the nearest neighbor in the 68 facial-landmarks space.

Original: We measure the similarity between I j and G θ ( c i , s k ) .
Revised: We measure the similarity between I j and G θ ( c i , s k ) .

Original: A. N ETWORK S TRUCTURE
Revised: A. N ETWORK S TRUCTURE

Original: Our Single C-S Fusion framework and Multiple C-S Fusion framework are shown in Figure 8.
Revised: Our Single C-S Fusion framework and Multiple C-S Fusion framework are shown in Figure 8.

Original: For the details of the reparametric module R , please refer to Appendix C.2.
Revised: For the details of the reparametric module R , please refer to Appendix C.2.

Original: F C
Revised: F C

Original: C o n v×
Revised: C o n v×

Original: N o r m a li z
Revised: N o r m a li z

Original: a t i o n × + F C
Revised: a t i o n × + F C

Original: F C
Revised: F C

Original: FC FC
Revised: FC FC

Original: C o n v×
Revised: C o n v×

Original: C o n v×𝑠𝑠𝑠 N …
Revised: C o n v×𝑠𝑠𝑠 N …

Original: Reparametric module R c𝑐𝑐 N …
Revised: Reparametric module R c𝑐𝑐 N …

Original: Reparametric module R መ𝐼
Revised: Reparametric module R መ𝐼

Original: [Figure]
Revised: [Figure]

Original: U p s a m p li n g U p s a m p li n g
Revised: U p s a m p li n g U p s a m p li n g

Original: U p s a m p li n g
Revised: U p s a m p li n g

Original: C - S Fusion Block
Revised: C - S Fusion Block

Original: Generator
Revised: Generator

Original: [Figure]
Revised: [Figure]

Original: F C F C
Revised: F C F C

Original: F C 𝑠𝑠𝑠 N …
Revised: F C 𝑠𝑠𝑠 N …

Original: Reparametricmodule R c𝑐𝑐 N …
Revised: Reparametricmodule R c𝑐𝑐 N …

Original: Reparametricmodule R
Revised: Reparametricmodule R

Original: C o n v×
Revised: C o n v×

Original: C o n v×መ𝐼
Revised: C o n v×መ𝐼

Original: [Figure]
Revised: [Figure]

Original: C o n v×
Revised: C o n v×

Original: C - S F u s i o n
Revised: C - S F u s i o n

Original: U p s a m p li n g
Revised: U p s a m p li n g

Original: C - S F u s i o n
Revised: C - S F u s i o n

Original: U p s a m p li n g
Revised: U p s a m p li n g

Original: C - S F u s i o n
Revised: C - S F u s i o n

Original: U p s a m p li n g
Revised: U p s a m p li n g

Original: C - S F u s i o n
Revised: C - S F u s i o n

Original: N o r m a li z
Revised: N o r m a li z

Original: a t i o n × +
Revised: a t i o n × +

Original: FC FC
Revised: FC FC

Original: C -S Fusion Block
Revised: C -S Fusion Block

Original: Generator
Revised: Generator

Original: [Figure]
Revised: [Figure]

Original: B B ASELINE D ETAILS
Revised: B B ASELINE D ETAILS

Original: For the datasets in the main paper, Car3D contains 183 car models, each rendered from 96 poses.
Revised: For the datasets in the main paper, Car3D contains 183 car models, each rendered from 96 poses.

Original: Chairs consists of 1393 chair models, each rendered from 62 poses.
Revised: Chairs consists of 1393 chair models, each rendered from 62 poses.

Original: CelebA contains 202,599 facial images of 10,177 celebrities.
Revised: CelebA contains 202,599 facial images of 10,177 celebrities.

Original: Market-1501 have 25,259 images.
Revised: Market-1501 have 25,259 images.

Original: For the baselines, we use open-sourced implementations for Cycle-VAE (Jha et al., 2018) 2 , DrNet (Denton & Birodkar, 2017) 3 , Lord (Gabbay & Hoshen, 2020) 4 and FactorVAE (Kim & Mnih, 2018) 5 .
Revised: For the baselines, we use open-sourced implementations for Cycle-VAE (Jha et al., 2018) 2 , DrNet (Denton & Birodkar, 2017) 3 , Lord (Gabbay & Hoshen, 2020) 4 and FactorVAE (Kim & Mnih, 2018) 5 .

Original: For FactorVAE, we traverse the latent space to select the dimensions related to pose as content embedding and treat the other dimensions as style embedding.
Revised: For FactorVAE, we traverse the latent space to select the dimensions related to pose as content embedding and treat the other dimensions as style embedding.

Original: For Wu et al. (2019c), there is no opensourced implementation.
Revised: For Wu et al. (2019c), there is no opensourced implementation.

Original: We use the code from https://github.com/CompVis/vunet , which uses ground truth landmarks as input instead of learning the landmarks unsupervisedly.
Revised: We use the code from https://github.com/CompVis/vunet , which uses ground truth landmarks as input instead of learning the landmarks unsupervisedly.

Original: To achieve the pseudo ground truth landmarks, we use the face detection library (Bulat & Tzimiropoulos, 2017) for Celeba.
Revised: To achieve the pseudo ground truth landmarks, we use the face detection library (Bulat & Tzimiropoulos, 2017) for Celeba.

Original: We try to use the L1 and perceptual loss for all the baselines and select the best.
Revised: We try to use the L1 and perceptual loss for all the baselines and select the best.

Original: We split the datasets into training and testing sets.
Revised: We split the datasets into training and testing sets.

Original: For Celeba, we randomly select 1000 among 10177 celebrities for testing.
Revised: For Celeba, we randomly select 1000 among 10177 celebrities for testing.

Original: For Car3D, we randomly select 20 among 183 CAD models for testing.
Revised: For Car3D, we randomly select 20 among 183 CAD models for testing.

Original: For Chairs, we randomly select 100 among 1393 models for testing.
Revised: For Chairs, we randomly select 100 among 1393 models for testing.

Original: For baselines with group supervision, only the training sets are used for training.
Revised: For baselines with group supervision, only the training sets are used for training.

Original: For unsupervised baselines and our method, all the datasets are used for training.
Revised: For unsupervised baselines and our method, all the datasets are used for training.

Original: C T ECHNICAL C OMPONENTS
Revised: C T ECHNICAL C OMPONENTS

Original: Here we present three technical components that are helpful to the C-S disentanglement.
Revised: Here we present three technical components that are helpful to the C-S disentanglement.

Original: The ablation study for these components is shown in Appendix E.
Revised: The ablation study for these components is shown in Appendix E.

Original: C. L ATENT O PTIMIZATION .
Revised: C. L ATENT O PTIMIZATION .

Original: In the C-S disentanglement literature, it is common to use encoders to predict embeddings, while latent optimization (Bojanowski et al., 2018; Gabbay & Hoshen, 2020) directly optimizes the embeddings via back-propagating without using encoders.
Revised: In the C-S disentanglement literature, it is common to use encoders to predict embeddings, while latent optimization (Bojanowski et al., 2018; Gabbay & Hoshen, 2020) directly optimizes the embeddings via back-propagating without using encoders.

Original: Encoders have a large number of parameters and require a lot more extra effort for training.
Revised: Encoders have a large number of parameters and require a lot more extra effort for training.

Original: Therefore, We adopt the latent optimization approach to update the latent spaces directly.
Revised: Therefore, We adopt the latent optimization approach to update the latent spaces directly.

Original: C. R EPARAMETRIC M ODULE
Revised: C. R EPARAMETRIC M ODULE

Original: Inspired by VAE (Kingma & Welling, 2014), we design a reparametric module to force the latent space to be continuous.
Revised: Inspired by VAE (Kingma & Welling, 2014), we design a reparametric module to force the latent space to be continuous.

Original: Thus, the embeddings encoding similar information will get closer in the latent space.
Revised: Thus, the embeddings encoding similar information will get closer in the latent space.

Original: Assume we have a mean embedding µ with a standard deviation σ , the reparametrized output is σX + µ , where X ∼ N (0 , I ) .
Revised: Assume we have a mean embedding µ with a standard deviation σ , the reparametrized output is σX + µ , where X ∼ N (0 , I ) .

Original: To further simplify the problem, we set σ = 1 following
Revised: To further simplify the problem, we set σ = 1 following

Original: Wu et al. (2019c) and Gabbay & Hoshen (2020).
Revised: Wu et al. (2019c) and Gabbay & Hoshen (2020).

Original: The mean embedding is the input style or content embedding.
Revised: The mean embedding is the input style or content embedding.

Original: The reparametric module can make the latent space continuous, which is helpful for backpropagation.
Revised: The reparametric module can make the latent space continuous, which is helpful for backpropagation.

Original: Though the training images have discrete identities, the optimized style embedding space is continuous.
Revised: Though the training images have discrete identities, the optimized style embedding space is continuous.

Original: Different style embeddings of the people with similar appearances are close to each other, as shown in Figure 4 (c).
Revised: Different style embeddings of the people with similar appearances are close to each other, as shown in Figure 4 (c).

Original: C. I NSTANCE DISCRIMINATION LOSS
Revised: C. I NSTANCE DISCRIMINATION LOSS

Original: We ﬁrst pretrain a ResNet-18 (He et al., 2016)
Revised: We ﬁrst pretrain a ResNet-18 (He et al., 2016)

Original: Φ and deﬁne a collection of layers of Φ as { Φ l } .
Revised: Φ and deﬁne a collection of layers of Φ as { Φ l } .

Original: Among several representative methods (Wu et al., 2018; Ye et al., 2019; He et al., 2020), we observe that the method in Wu et al.
Revised: Among several representative methods (Wu et al., 2018; Ye et al., 2019; He et al., 2020), we observe that the method in Wu et al.

Original: (2018) achieves the best performance in our task.
Revised: (2018) achieves the best performance in our task.

Original: Given two images I i and I j , we mix the embeddings to generate u =
Revised: Given two images I i and I j , we mix the embeddings to generate u =

Original: G
Revised: G

Original: ( R ( s i ) , R ( c j )) and v =
Revised: ( R ( s i ) , R ( c j )) and v =

Original: G ( R ( s j ) , R ( c i )) .
Revised: G ( R ( s j ) , R ( c i )) .

Original: For samples sharing the same style embedding, we enforce the feature distance in Φ between them to be close.
Revised: For samples sharing the same style embedding, we enforce the feature distance in Φ between them to be close.

Original: This loss term can be written as
Revised: This loss term can be written as

Original: [Equation]
Revised: [Equation]

Original: where x = G ( R ( s i ) , R ( c i )) and y =
Revised: where x = G ( R ( s i ) , R ( c i )) and y =

Original: G ( R ( s j ) , R ( c j )) .
Revised: G ( R ( s j ) , R ( c j )) .

Original: The hyperparameters { λ l } balance the contribution of each layer l to the loss.
Revised: The hyperparameters { λ l } balance the contribution of each layer l to the loss.

Original: { λ l } are set to be [1 , 1 , 1 , 1 , 1]
Revised: { λ l } are set to be [1 , 1 , 1 , 1 , 1]

Original: C. I NFORMATION BOTTLENECK
Revised: C. I NFORMATION BOTTLENECK

Original: Similar to Anneal VAE (Burgess et al., 2018), we introduce a information bottleneck given by
Revised: Similar to Anneal VAE (Burgess et al., 2018), we introduce a information bottleneck given by

Original: [Equation]
Revised: [Equation]

Original: (7) where C s and C c are the information capacity controlling the amount of information of the content and style respectively.
Revised: (7) where C s and C c are the information capacity controlling the amount of information of the content and style respectively.

Original: During training, C s and C c increase linearly.
Revised: During training, C s and C c increase linearly.

Original: The rate of increase is controlled by the increase steps and the maximum value.
Revised: The rate of increase is controlled by the increase steps and the maximum value.

Original: By controlling the increase rate, the content is forced to encode information ﬁrst, so that the learning process is more consistent with our assumptions about the data: the shared conditional variable c is learned ﬁrst.
Revised: By controlling the increase rate, the content is forced to encode information ﬁrst, so that the learning process is more consistent with our assumptions about the data: the shared conditional variable c is learned ﬁrst.

Original: For the information bottleneck, by taking the training process of the model without the information bottleneck as a reference, we determine the increase steps and the maximum of the information capacity C c and C s .
Revised: For the information bottleneck, by taking the training process of the model without the information bottleneck as a reference, we determine the increase steps and the maximum of the information capacity C c and C s .

Original: We can enhance the model inductive bias by tuning these parameters.
Revised: We can enhance the model inductive bias by tuning these parameters.

Original: For
Revised: For

Original: Chairs, we set the maximum of C c to 5, the start value of C c to 2 , the increase steps of C c to 1 .
Revised: Chairs, we set the maximum of C c to 5, the start value of C c to 2 , the increase steps of C c to 1 .

Original: 4 × 10 5 , γ c to 1 and γ s to 0.
Revised: 4 × 10 5 , γ c to 1 and γ s to 0.

Original: Note that our model achieves state-of-the-art performance on Chairs even without information bottleneck.
Revised: Note that our model achieves state-of-the-art performance on Chairs even without information bottleneck.

Original: D M ORE RESULTS
Revised: D M ORE RESULTS

Original: In this section, we demonstrate more qualitative comparison and more qualitative results (including more datasets).
Revised: In this section, we demonstrate more qualitative comparison and more qualitative results (including more datasets).

Original: D. M ORE QUALITATIVE EXPERIMENTS
Revised: D. M ORE QUALITATIVE EXPERIMENTS

Original: In the main paper, for unsupervised baselines, we only compare our method with FactorVAE (Kim & Mnih, 2018) limited to space.
Revised: In the main paper, for unsupervised baselines, we only compare our method with FactorVAE (Kim & Mnih, 2018) limited to space.

Original: As shown in Figure 9, we also outperform Wu et al. (2019c).
Revised: As shown in Figure 9, we also outperform Wu et al. (2019c).

Original: For
Revised: For

Original: Wu et al. (2019c), the disentanglement is poor, such that the content embeddings control almost all the factors while the style embeddings control the tone.
Revised: Wu et al. (2019c), the disentanglement is poor, such that the content embeddings control almost all the factors while the style embeddings control the tone.

Original: For datsets in the main paper, We provide more qualitative results in Fig.
Revised: For datsets in the main paper, We provide more qualitative results in Fig.

Original: 24, 25, 26, 27 and 28.
Revised: 24, 25, 26, 27 and 28.

Original: Moreover, we also apply our method on higher resolution images and achieve good performance, as shown in Figure 20.
Revised: Moreover, we also apply our method on higher resolution images and achieve good performance, as shown in Figure 20.

Original: D. M ORE DATASETS
Revised: D. M ORE DATASETS

Original: Besides the datasets introduced in the main paper, we make additional experiments on other datasets: such as MNIST (LeCun et al., 2010), Cat (Parkhi et al., 2012; Zhang et al., 2008), Anime (Chao, 2019).
Revised: Besides the datasets introduced in the main paper, we make additional experiments on other datasets: such as MNIST (LeCun et al., 2010), Cat (Parkhi et al., 2012; Zhang et al., 2008), Anime (Chao, 2019).

Original: MNIST has 70k examples for 10 handwritten digits.
Revised: MNIST has 70k examples for 10 handwritten digits.

Original: Cat has 1.2k cat head images.
Revised: Cat has 1.2k cat head images.

Original: Anime contains 63,632 anime faces.
Revised: Anime contains 63,632 anime faces.

Original: The results are shown in Figure 21, 22 ,23 and 21.
Revised: The results are shown in Figure 21, 22 ,23 and 21.

Original: [Figure]
Revised: [Figure]

Original: D. M ORE 3D RECONSTRUCTION
Revised: D. M ORE 3D RECONSTRUCTION

Original: Our setting treats every image as a single identity (style) without ambiguity for augmenting singleview images.
Revised: Our setting treats every image as a single identity (style) without ambiguity for augmenting singleview images.

Original: On Celeba, We use MVF-Net (Wu et al., 2019a) based on multi-view to reconstruct 3D facial shapes.
Revised: On Celeba, We use MVF-Net (Wu et al., 2019a) based on multi-view to reconstruct 3D facial shapes.

Original: For a given image, we can get the corresponding style embedding content embedding.
Revised: For a given image, we can get the corresponding style embedding content embedding.

Original: Then we can get the front, left, and right view of this image combining the extracted style embedding
Revised: Then we can get the front, left, and right view of this image combining the extracted style embedding

Original: [Figure]
Revised: [Figure]

Original: and prepared content embeddings 6 .
Revised: and prepared content embeddings 6 .

Original: As shown in Figure 11, our augmented multi-view images are consistent, and the 3D meshes based on our method are more accurate than those based on Lord.
Revised: As shown in Figure 11, our augmented multi-view images are consistent, and the 3D meshes based on our method are more accurate than those based on Lord.

Original: E M ORE ABLATION STUDY
Revised: E M ORE ABLATION STUDY

Original: Here we perform more ablation study for the technical modules.
Revised: Here we perform more ablation study for the technical modules.

Original: If we use an amortized scheme instead of a latent optimization scheme, there are leaks between style and content latent space, and the result is worse than latent optimization, as shown in Figure 10 (a) and (c).
Revised: If we use an amortized scheme instead of a latent optimization scheme, there are leaks between style and content latent space, and the result is worse than latent optimization, as shown in Figure 10 (a) and (c).

Original: Furthermore, if we do not use a reparametric module, we ﬁnd the reconstruction performance is worse, as shown in Figure 10 (b).
Revised: Furthermore, if we do not use a reparametric module, we ﬁnd the reconstruction performance is worse, as shown in Figure 10 (b).

Original: For the instance discrimination loss, the comparison is shown in Table 4.
Revised: For the instance discrimination loss, the comparison is shown in Table 4.

Original: The disentanglement is better with an instance discrimination loss.
Revised: The disentanglement is better with an instance discrimination loss.

Original: For the information bottleneck, as shown in Table 3, the result with an information bottleneck is much better than the one without it.
Revised: For the information bottleneck, as shown in Table 3, the result with an information bottleneck is much better than the one without it.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: F T RAINING VISUALIZATION
Revised: F T RAINING VISUALIZATION

Original: To better understand how our design helps to guide the disentanglement, we visualize the generated images during the training process in Figure 12.
Revised: To better understand how our design helps to guide the disentanglement, we visualize the generated images during the training process in Figure 12.

Original: As the generated images show, a mean shape of faces is ﬁrst learned.
Revised: As the generated images show, a mean shape of faces is ﬁrst learned.

Original: Then the faces start to rotate, which indicates the pose is disentangled to the content space.
Revised: Then the faces start to rotate, which indicates the pose is disentangled to the content space.

Original: After that, the identity features emerge as the style starts to learn parameters for customizing the shared distribution to approximate the real faces distribution.
Revised: After that, the identity features emerge as the style starts to learn parameters for customizing the shared distribution to approximate the real faces distribution.

Original: [Figure]
Revised: [Figure]

Original: G U NSEEN I MAGES I NFERENCE
Revised: G U NSEEN I MAGES I NFERENCE

Original: Though we learn to disentangle in an unsupervised manner, we may need to process unseen images.
Revised: Though we learn to disentangle in an unsupervised manner, we may need to process unseen images.

Original: An natural solution is to train encoders to encode images to the latent spaces.
Revised: An natural solution is to train encoders to encode images to the latent spaces.

Original: We train style encoder
Revised: We train style encoder

Original: E s and content encoder E c
Revised: E s and content encoder E c

Original: by
Revised: by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: After training, we applied our model trained on the CelebA dataset to faces collected by Wu et al.
Revised: After training, we applied our model trained on the CelebA dataset to faces collected by Wu et al.

Original: (2020) including paintings and cartoon drawings.
Revised: (2020) including paintings and cartoon drawings.

Original: As shown in Figure 13, our method still works well even though it has never seen such images during training.
Revised: As shown in Figure 13, our method still works well even though it has never seen such images during training.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: H C OMPARISON WITH SELECTED RELATED WORK
Revised: H C OMPARISON WITH SELECTED RELATED WORK

Original: Comparison with StyleGAN.
Revised: Comparison with StyleGAN.

Original: In our framework, the optimized content (conv) and style embeddings are disentangled representations of corresponding images.
Revised: In our framework, the optimized content (conv) and style embeddings are disentangled representations of corresponding images.

Original: While StyleGAN (Karras et al., 2019) keeps the input of the convolution branch as a learnt constant for the whole dataset and ﬁnds the feature space of the “style” branch has disentanglement ability.
Revised: While StyleGAN (Karras et al., 2019) keeps the input of the convolution branch as a learnt constant for the whole dataset and ﬁnds the feature space of the “style” branch has disentanglement ability.

Original: For StyleGAN2 (Karras et al., 2020) 7 , we select the subset of “style”, which represents pose, as the content embedding and the rest subset as the style embedding.
Revised: For StyleGAN2 (Karras et al., 2020) 7 , we select the subset of “style”, which represents pose, as the content embedding and the rest subset as the style embedding.

Original: As shown in Figure 15, StyleGAN2 entangled pose with other semantic attributes, such as hair and glasses.
Revised: As shown in Figure G, StyleGAN2 entangled pose with other semantic attributes, such as hair and glasses.

Original: As shown in Figure 28, the content of our method on human faces is pose attribute without entanglement.
Revised: As shown in Figure 28, the content of our method on human faces is pose attribute without entanglement.

Original: Comparsion with MUNIT & Park et al.
Revised: Comparsion with MUNIT & Park et al.

Original: Starting with our assumption that content embeddings share the same distribution, and leveraging AdaIN-like operation, we achieve unsupervised content and style disentanglement without needing “swapping” operation and GAN loss constraint to extract the shared content information as image translation works (MUNIT (Huang et al., 2018b) and Park et al.
Revised: Starting with our assumption that content embeddings share the same distribution, and leveraging AdaIN-like operation, we achieve unsupervised content and style disentanglement without needing “swapping” operation and GAN loss constraint to extract the shared content information as image translation works (MUNIT (Huang et al., 2018b) and Park et al.

Original: do.
Revised: do.

Original: As shown in Figure 14, for MUNIT (Huang et al., 2018b) and Park et al.
Revised: As shown in Figure G, for MUNIT (Huang et al., 2018b) and Park et al.

Original: (2020), the content is low-level structure information.
Revised: (2020), the content is low-level structure information.

Original: While in our case, the content is a high-level semantic attribute of the object, e.g., the pose attribute.
Revised: While in our case, the content is a high-level semantic attribute of the object, e.g., the pose attribute.

Original: As shown in Figure 14 (d), we can also achieve the similar performance to exchange the tone of the images by exchanging the ﬁne style.
Revised: As shown in Figure G (d), we can also achieve the similar performance to exchange the tone of the images by exchanging the ﬁne style.

Original: The ﬁne styles in our method are the style inputs of the last C-S fusion block in the multiple C-S fusion framework,
Revised: The ﬁne styles in our method are the style inputs of the last C-S fusion block in the multiple C-S fusion framework.

Original: I C ROSS - DOMAIN A PPLICATION
Revised: I C ROSS - DOMAIN A PPLICATION

Original: As shown in the main paper, the content and style are disentangled in a single domain.
Revised: As shown in the main paper, the content and style are disentangled in a single domain.

Original: Based on our assumption, the cross-domain dataset also can be disentangled.
Revised: Based on our assumption, the cross-domain dataset also can be disentangled.

Original: In this section, we test our model on a cross-domain dataset to further verify our assumption.
Revised: In this section, we test our model on a cross-domain dataset to further verify our assumption.

Original: In some cases that we merge images from two domains, our method can still work and achieve performance, which is similar to domain translation.
Revised: In some cases that we merge images from two domains, our method can still work and achieve performance, which is similar to domain translation.

Original: For example, Edges2Shoes (Yu & Grauman, 2014) is a dataset consisting of 50k paired shoe and edge map images.
Revised: For example, Edges2Shoes (Yu & Grauman, 2014) is a dataset consisting of 50k paired shoe and edge map images.

Original: As shown in Figure 16, the content is edge structure, and the style is texture.
Revised: As shown in Figure 16, the content is edge structure, and the style is texture.

Original: Thanks to this, we can translate edge images into shoe images and vice versa without any additional operation.
Revised: Thanks to this, we can translate edge images into shoe images and vice versa without any additional operation.

Original: Furthermore, once the domain labels are given, we can disentangle and align the cross-domain dataset.
Revised: Furthermore, once the domain labels are given, we can disentangle and align the cross-domain dataset.

Original: This experiment may be helpful for domain transfer and domain adaptation.
Revised: This experiment may be helpful for domain transfer and domain adaptation.

Original: We train our model on the dataset that consists of Celeba and Anime.
Revised: We train our model on the dataset that consists of Celeba and Anime.

Original: The model needs to be modiﬁed for learning cross-domain data: concatenate the domain embedding and the style embedding, take it as the style embedding in the original model, and optimize the domain embedding during latent optimization.
Revised: The model needs to be modiﬁed for learning cross-domain data: concatenate the domain embedding and the style embedding, take it as the style embedding in the original model, and optimize the domain embedding during latent optimization.

Original: The results are shown in Figure 17.
Revised: The results are shown in Figure 17.

Original: The learned poses are well aligned both in the animation and reality domain.
Revised: The learned poses are well aligned both in the animation and reality domain.

Original: the size of the style embedding d s to 256, the size of the content embedding d c to

Original: The content embedding are well aligned in domain

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: J P ROOF
Revised: J P ROOF

Original: Our optimization target is to minimize the KL divergence between P and Q,
Revised: Our optimization target is to minimize the KL divergence between P and Q,

Original: [Equation]
Revised: [Equation]

Original: Expanding the above KL term
Revised: Expanding the above KL term

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: The above integral equation cannot be directly calculated
Revised: The above integral equation cannot be directly calculated

Original: [Equation]
Revised: [Equation]

Original: but can be estimated by the sampled images { I j}
Revised: but can be estimated by the sampled images { I j

Original: [Equation]

Original: P i

Original: [Equation]
Revised: [Equation]

Original: Separating P and Q from the above equation by logarithmic transformation
Revised: Separating P and Q from the above equation by logarithmic transformation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Since P i ( x = I j | c = c i ) is the dataset distribution [Equation] which is an unknown constant distribution, therefore, the ﬁrst term is a constant [Equation]
Revised: Since P i ( x = I j | c = c i ) is the dataset distribution, which is an unknown constant distribution, therefore, the ﬁrst term is a constant, the optimization target is equivalent to

Original: the optimization target is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Rewriting it into mathematical expectation form, we have
Revised: Rewriting it into mathematical expectation form, we have

Original: [Equation]
Revised: [Equation]

Original: where P i refers to P i ( x = I j | c = c i ) .
Revised: where P i refers to P i ( x = I j | c = c i ) .

Original: Our optimization target is equivalent to maximum likelihood estimation.
Revised: Our optimization target is equivalent to maximum likelihood estimation.

Original: Here we assume Q is a Gaussian distribution,
Revised: Here we assume Q is a Gaussian distribution,

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: 14 and Eq.
Revised: 14 and Eq.

Revised: [Equation]

Original: 15, we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Consequently, the ﬁnal optimization target is
Revised: Consequently, the ﬁnal optimization target is

Original: [Equation]
Revised: [Equation]

Original: Q.E.D.