Original: E MERGING D ISENTANGLEMENT IN A UTO -E NCODER B ASED U NSUPERVISED I MAGE C ONTENT T RANSFER
Revised: E MERGING D ISENTANGLEMENT IN A UTO -E NCODER B ASED U NSUPERVISED I MAGE C ONTENT T RANSFER

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the problem of learning to map, in an unsupervised way, between domains A and B , such that the samples b ∈ B contain all the information that exists in samples a ∈ A and some additional information.
Revised: We study the problem of learning to map, in an unsupervised way, between domains A and B , such that the samples b ∈ B contain all the information that exists in samples a ∈ A and some additional information.

Original: For example, ignoring occlusions, B can be people with glasses, A people without, and the glasses, would be the added information.
Revised: For example, ignoring occlusions, B can be people with glasses, A people without, and the glasses, would be the added information.

Original: When mapping a sample a from the ﬁrst domain to the other domain, the missing information is replicated from an independent reference sample b ∈ B .
Revised: When mapping a sample a from the ﬁrst domain to the other domain, the missing information is replicated from an independent reference sample b ∈ B .

Original: Thus, in the above example, we can create, for every person without glasses a version with the glasses observed in any face image.
Revised: Thus, in the above example, we can create, for every person without glasses a version with the glasses observed in any face image.

Original: Our solution employs a single two-pathway encoder and a single decoder for both domains.
Revised: Our solution employs a single two-pathway encoder and a single decoder for both domains.

Original: The common part of the two domains and the separate part are encoded as two vectors, and the separate part is ﬁxed at zero for domain A .
Revised: The common part of the two domains and the separate part are encoded as two vectors and the separate part is ﬁxed at zero for domain A .

Original: The loss terms are minimal and involve reconstruction losses for the two domains and a domain confusion term.
Revised: The loss terms are minimal and involve reconstruction losses for the two domains and a domain confusion term.

Original: Our analysis shows that under mild assumptions, this architecture, which is much simpler than the literature guided-translation methods, is enough to ensure disentanglement between the two domains.
Revised: Our analysis shows that under mild assumptions, this architecture, which is much simpler than the literature guided-translation methods, is enough to ensure disentanglement between the two domains.

Original: We present convincing results in a few visual domains, such as no-glasses to glasses, adding facial hair based on a reference image, etc.
Revised: We present convincing results in a few visual domains, such as no-glasses to glasses, adding facial hair based on a reference image, etc

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In the problem of unsupervised domain translation, the algorithm receives two sets of samples, one from each domain, and learns a function that maps between a sample in one domain to the analogous sample in the other domain (Zhu et al., 2017a;
Revised: In the problem of unsupervised domain translation, the algorithm receives two sets of samples, one from each domain, and learns a function that maps between a sample in one domain to the analogous sample in the other domain (Zhu et al., 2017a;

Original: Kim et al., 2017; Yi et al., 2017; Benaim & Wolf, 2017; Liu & Tuzel, 2016;
Revised: Kim et al., 2017; Yi et al., 2017; Benaim & Wolf, 2017; Liu & Tuzel, 2016;

Original: Liu et al., 2017; Choi et al., 2017; Conneau et al., 2017; Zhang et al., 2017a;b; Lample et al., 2018).
Revised: Liu et al., 2017; Choi et al., 2017; Conneau et al., 2017; Zhang et al., 2017a;b; Lample et al., 2018).

Original: The term unsupervised means, in this context, that the two sets are unpaired.
Revised: The term unsupervised means, in this context, that the two sets are unpaired.

Original: In this paper, we consider the problem of domain B , which contains a type of content that is not present at A .
Revised: In this paper, we consider the problem of domain B , which contains a type of content that is not present at A .

Original: As a running example, we consider the problem of mapping between a face without eyewear (domain A ) to a face with glasses (domain B ).
Revised: As a running example, we consider the problem of mapping between a face without eyewear (domain A ) to a face with glasses (domain B ).

Original: While most methods would map to a person with any glasses, our solution is guided and we attach to an image a ∈ A , the glasses that are present in a reference image b ∈ B .
Revised: While most methods would map to a person with any glasses, our solution is guided and we attach to an image a ∈ A , the glasses that are present in a reference image b ∈ B .

Original: In comparison to other guided image to image translation methods, our method is considerably simpler.
Revised: In comparison to other guided image to image translation methods, our method is considerably simpler.

Original: It relies on having a latent space with two parts: (i) a shared part that is common to both
Revised: It relies on having a latent space with two parts: (i) a shared part that is common to both

Original: A and B , and (ii) a speciﬁc part that encodes the added content in B .
Revised: A and B , and (ii) a speciﬁc part that encodes the added content in B .

Original: By setting the second part to be the zero vector for all samples in A , a disentanglement emerges.
Revised: By setting the second part to be the zero vector for all samples in A , a disentanglement emerges.

Original: Our analysis shows that thisleads to the ability to train with a simple, straightforward domain confusion term, while enjoying the generalization guarantees that would otherwise require a more elaborate loss. 1. P REVIOUS W ORK In image to image translation, the transformations are often captured by multi-layered networks of an encoder-decoder architecture. The existing solutions often assume a one to one mapping between the domains, i.e., that there exists a function y such that given a sample a in domain A , maps it to an analog sample in domain B .
Revised: Our analysis shows that this

Original: In fact, the circularity based constraints by Zhu et al.

Original: (2017a)
Revised: [Table]

Original: ; Kim et al. ; Yi et al. (2017) are based on this assumption, since going from one domain to the other MUNIT (Huang, 2018)
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: DRIT (Lee, 2018)

Original: [Table]
Revised: [Table]

Original: (Chang’18)
Revised: (Chang’18)

Original: Our

Original: [Table]
Revised: [Table]

Original: h a r i
Revised: h a r i

Original: [Table]
Revised: [Table]

Original: p a tt e r n
Revised: p a tt e r n

Original: [Table]
Revised: [Table]

Original: + +
Revised: + +

Original: [Table]
Revised: [Table]

Original: + + +
Revised: + + +

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: N u m b e r o f n e t w 
Revised: N u m b e r o f n e t w o r k s

Original: [Table]
Revised: [Table]

Revised: leads to the ability to train with a simple, straightforward domain confusion term, while enjoying the generalization guarantees that would otherwise require a more elaborate loss.

Revised: 1. P REVIOUS W ORK

Revised: In image to image translation, the transformations are often captured by multi-layered networks of an encoder-decoder architecture.

Revised: The existing solutions often assume a one to one mapping between the domains, i.e., that there exists a function y such that given a sample a in domain A , maps it to an analog sample in domain B .

Revised: In fact, the circularity based constraints by Zhu et al.

Original:  and back, it is assumed that the original sample is obtained, which requires no loss of information.
Revised: (2017a) ; Kim et al. ; Yi et al. (2017) are based on this assumption, since going from one domain to the other and back, it is assumed that the original sample is obtained, which requires no loss of information.

Original: However, to employ an example made popular by Zhu et al.
Revised: However, to employ an example made popular by Zhu et al.

Original: (2017a), when going from a zebra to a horse, the stripes are lost, which results in an ambiguity when mapping in the other direction.
Revised: (2017a), when going from a zebra to a horse, the stripes are lost, which results in an ambiguity when mapping in the other direction.

Original: This shortcoming was identiﬁed by the literature, and a few contributions present many to many mappings.
Revised: This shortcoming was identiﬁed by the literature, and a few contributions present many to many mappings.

Original: These include the augmented CycleGAN Almahairi et al.
Revised: These include the augmented CycleGAN Almahairi et al.

Original: (2018), which adds a random vector to each domain.
Revised: (2018), which adds a random vector to each domain.

Original: A completely different approach is taken by the NAM method (Hoshen &
Revised: A completely different approach is taken by the NAM method (Hoshen &

Original: Wolf, 2018), in which multiple solutions are obtained by considering multiple random initializations.
Revised: Wolf, 2018), in which multiple solutions are obtained by considering multiple random initializations.

Original: In our work, multiplicity of outcomes arises from using a reference (guide) image.
Revised: In our work, multiplicity of outcomes arises from using a reference (guide) image.

Original: A powerful way to capture relations between the two domains, is by employing separate autoencoders for the two domains, but which share many of the weights (Liu & Tuzel, 2016; Liu et al., 2017).
Revised: A powerful way to capture relations between the two domains, is by employing separate autoencoders for the two domains, but which share many of the weights (Liu & Tuzel, 2016; Liu et al., 2017).

Original: This leads to a shared representation: while the low-level image properties, such as texture and color, are domain-speciﬁc and encoded/decoded separately, the mid- and top-level properties are common to both domains and are processed by identical replicas of the same layers.
Revised: This leads to a shared representation: while the low-level image properties, such as texture and color, are domain-speciﬁc and encoded/decoded separately, the mid- and top-level properties are common to both domains and are processed by identical replicas of the same layers.

Original: In our work, we rely on a shared encoder for both domains in order to enforce a shared representation.
Revised: In our work, we rely on a shared encoder for both domains in order to enforce a shared representation.

Original: Our method performs one-sided mapping, from A to B , and does not learn the mapping in the other direction.
Revised: Our method performs one-sided mapping, from A to B , and does not learn the mapping in the other direction.

Original: Benaim & Wolf (2017) perform one-sided mapping using a distance based constraint, which we do not employ.
Revised: Benaim & Wolf (2017) perform one-sided mapping using a distance based constraint, which we do not employ.

Original: That method is symmetric in the two domains and in many experiments, the distance constraint is used in tandem with the cycle constraint.
Revised: That method is symmetric in the two domains and in many experiments, the distance constraint is used in tandem with the cycle constraint.

Original: In our case, the two domains are asymmetric and one domain contains an added content.
Revised: In our case, the two domains are asymmetric and one domain contains an added content.

Original: Our method is inherently asymmetric, reﬂecting the asymmetry between the source image and the guide image, which contains the additional content.
Revised: Our method is inherently asymmetric, reﬂecting the asymmetry between the source image and the guide image, which contains the additional content.

Original: The work by Taigman et al.
Revised: The work by Taigman et al.

Original: (2017); Hoshen & Wolf (2018) map in an asymmetric way, but rely on the existence of a perceptual distance, which we do not employ.
Revised: (2017); Hoshen & Wolf (2018) map in an asymmetric way, but rely on the existence of a perceptual distance, which we do not employ.

Original: Guided Translation The most relevant work contains very recent and concurrent methods in which the mapping between the domains employs two inputs: a source image a
Revised: Guided Translation The most relevant work contains very recent and concurrent methods in which the mapping between the domains employs two inputs: a source image a

Original: and a reference (guide or attribute) image b .
Revised: and a reference (guide or attribute) image b .

Original: Tab. 1 compares these methods to ours along two axes: (i) where sharing occurs in the architecture, and (ii) the number of trained sub-networks of each type.
Revised: Tab. 1 compares these methods to ours along two axes: (i) where sharing occurs in the architecture, and (ii) the number of trained sub-networks of each type.

Original: The sharing can be of layers between different encoders and decoders, following, e.g., Liu & Tuzel (2016); sharing of a common part of a latent space; and using the same encoder or decoder for multiple domains.
Revised: The sharing can be of layers between different encoders and decoders, following, e.g., Liu & Tuzel (2016); sharing of a common part of a latent space; and using the same encoder or decoder for multiple domains.

Original: The networks are of four types: encoders, which map images to a latent space, generators (also known as decoders), which generate images from a latent representation, discriminators that are used as part of an adversarial loss, and other, less-standard, networks.
Revised: The networks are of four types: encoders, which map images to a latent space, generators (also known as decoders), which generate images from a latent representation, discriminators that are used as part of an adversarial loss, and other, less-standard, networks.

Original: It is apparent that our method is considerably simpler than the literature methods.
Revised: It is apparent that our method is considerably simpler than the literature methods.

Original: The main reason is that our method is based on the emergence of disentanglement, as detailed in Sec. 4.
Revised: The main reason is that our method is based on the emergence of disentanglement, as detailed in Sec. 4.

Original: This allows us to to train with many less parameters and without the need to apply excessive tuning, in order to balance or calibrate the various components of the compound loss.
Revised: This allows us to to train with many less parameters and without the need to apply excessive tuning, in order to balance or calibrate the various components of the compound loss.

Original: The MUNIT architecture by Huang et al.
Revised: The MUNIT architecture by Huang et al.

Original: (2018), like our architecture, employs a shared latent space, in addition to a domain speciﬁc latent space.
Revised: (2018), like our architecture, employs a shared latent space, in addition to a domain speciﬁc latent space.

Original: Their architecture is not limited to two domains 1 and unlike ours, employs separate encoders and decoders for the various domains.
Revised: Their architecture is not limited to two domains 1 and unlike ours, employs separate encoders and decoders for the various domains.

Original: The type of guiding that is obtained from the target domain in MUNIT is referred to as style, while in our case, the guidance provides content.
Revised: The type of guiding that is obtained from the target domain in MUNIT is referred to as style, while in our case, the guidance provides content.

Original: Therefore, MUNIT, as can be seen in our experiments, cannot add speciﬁc glasses, when shifting from the no-glasses domain to the faces with eyewear domain.
Revised: Therefore, MUNIT, as can be seen in our experiments, cannot add speciﬁc glasses, when shifting from the no-glasses domain to the faces with eyewear domain.

Original: The EG-UNIT architecture by Ma et al.
Revised: The EG-UNIT architecture by Ma et al.

Original: (2018) presents a few novelties, including an adaptive method of masking-out a varying set of the features in the shared latent space.
Revised: (2018) presents a few novelties, including an adaptive method of masking-out a varying set of the features in the shared latent space.

Original: In our latent representation of domain A , some of the features are constantly zero, which is much simpler.
Revised: In our latent representation of domain A , some of the features are constantly zero, which is much simpler.

Original: This method also focuses on guiding for style and not for content, as is apparent form their experiments.
Revised: This method also focuses on guiding for style and not for content, as is apparent form their experiments.

Original: The very recent DRIT work by Lee et al.
Revised: The very recent DRIT work by Lee et al.

Original: (2018) learns to map between two domains using a disentangled representation.
Revised: (2018) learns to map between two domains using a disentangled representation.

Original: Unlike our work, this work seems to focus on style rather than content.
Revised: Unlike our work, this work seems to focus on style rather than content.

Original: The proposed solution differs from us in many ways: (1) it relies on two-way mapping, while we only map from A to B .
Revised: The proposed solution differs from us in many ways: (1) it relies on two-way mapping, while we only map from A to B .

Original: (2) it relies on shared weights in order to ensure that the common representation is shared.
Revised: (2) it relies on shared weights in order to ensure that the common representation is shared.

Original: (3) it adds a VAE-like (Kingma & Welling, 2014) statistical characterization of the latent space, which results in the ability to sample random attributes.
Revised: (3) it adds a VAE-like (Kingma & Welling, 2014) statistical characterization of the latent space, which results in the ability to sample random attributes.

Original: As can be seen in Tab. 1, the solution of Lee et al.
Revised: As can be seen in Tab. 1, the solution of Lee et al.

Original: (2018) is considerably more involved than our solution.
Revised: (2018) is considerably more involved than our solution.

Original: DRIT (and also MUNIT) employ two different types of encoders that enforce a separation of the latent space representations to either style or content vectors.
Revised: DRIT (and also MUNIT) employ two different types of encoders that enforce a separation of the latent space representations to either style or content vectors.

Original: For example, the style encoder, unlike the content encoder, employs spatial pooling and it also results in a smaller representation than the content one.
Revised: For example, the style encoder, unlike the content encoder, employs spatial pooling and it also results in a smaller representation than the content one.

Original: This is important, in the context of these methods, in order to ensure that the two representations encode different aspects of the image.
Revised: This is important, in the context of these methods, in order to ensure that the two representations encode different aspects of the image.

Original: If DRIT or MUNIT were to use the same type of encoder twice, then one encoder could capture all the information, and the image-based guiding (mixing representations from two images) would become mute.
Revised: If DRIT or MUNIT were to use the same type of encoder twice, then one encoder could capture all the information, and the image-based guiding (mixing representations from two images) would become mute.

Original: In contrast, our method (i) does not separate style and content, and (ii) has a representation that is geared toward capturing the additional content.
Revised: In contrast, our method (i) does not separate style and content, and (ii) has a representation that is geared toward capturing the additional content.

Original: The work most similar to us in its goal, but not in method, is the PairedCycleGAN work by Chang et al.
Revised: The work most similar to us in its goal, but not in method, is the PairedCycleGAN work by Chang et al.

Original: This work explores the single application of applying the makeup of a reference face to a source face image.
Revised: This work explores the single application of applying the makeup of a reference face to a source face image.

Original: Unfortunately, the method was only demonstrated on a proprietary unshared dataset and the code is also not publicly available, making a direct comparison impossible at this time.
Revised: Unfortunately, the method was only demonstrated on a proprietary unshared dataset and the code is also not publicly available, making a direct comparison impossible at this time.

Original: The method itself is completely different from ours and does not employ disentanglement.
Revised: The method itself is completely different from ours and does not employ disentanglement.

Original: Instead, a generator with two image inputs is used to produce an output image, where the makeup is transfered between the input images, and a second generator is trained to remove makeup.
Revised: Instead, a generator with two image inputs is used to produce an output image, where the makeup is transfered between the input images, and a second generator is trained to remove makeup.

Original: The generation is done separately to k = 5 pre-segmented facial regions, and the generators do not employ an encoder-decoder architecture.
Revised: The generation is done separately to k = 5 pre-segmented facial regions, and the generators do not employ an encoder-decoder architecture.

Original: Lastly, there are guided methods, which are trained in the supervised domain, i.e., when there are matches between domain A and B .
Revised: Lastly, there are guided methods, which are trained in the supervised domain, i.e., when there are matches between domain A and B .

Original: Unlike the earlier one-to-one work, such as pix2pix Isola et al.
Revised: Unlike the earlier one-to-one work, such as pix2pix Isola et al.

Original: (2017b), these methods produce multiple outputs based on a reference image in the target domain.
Revised: (2017b), these methods produce multiple outputs based on a reference image in the target domain.

Original: Examples include the Bicycle GAN by Zhu et al.
Revised: Examples include the Bicycle GAN by Zhu et al.

Original: (2017b), who also applied, as baseline in their experiments, the methods of Bao et al.
Revised: (2017b), who also applied, as baseline in their experiments, the methods of Bao et al.

Original: (2017); Gonzalez-Garcia et al.
Revised: (2017); Gonzalez-Garcia et al.

Original: Other Disentanglement Work InfoGAN (Chen et al., 2016) learns a representation in which, due to the statistical properties of the representations, speciﬁc classes are encoded as a one-hot encoding of part of the latent vector.
Revised: Other Disentanglement Work InfoGAN (Chen et al., 2016) learns a representation in which, due to the statistical properties of the representations, speciﬁc classes are encoded as a one-hot encoding of part of the latent vector.

Original: In the work of Lample et al.
Revised: In the work of Lample et al.

Original: ; Hadad et al.
Revised: ; Hadad et al.

Original: , the representation is disentangled by reducing the class based information within it.
Revised: , the representation is disentangled by reducing the class based information within it.

Original: The separate class based information is different in nature from our multi-dimensional added content.
Revised: The separate class based information is different in nature from our multi-dimensional added content.

Original: Cao et al.
Revised: Cao et al.

Original: (2018), which builds upon Hadad et al.
Revised: (2018), which builds upon Hadad et al.

Original: (2018), performs guided image to image translation, but assumes the availability of class based information, which we do not.
Revised: (2018), performs guided image to image translation, but assumes the availability of class based information, which we do not.

Original:  P ROBLEM S ETUP
Revised:  P ROBLEM S ETUP

Original: We consider a setting with two domains A
Revised: We consider a setting with two domains A

Original: =
Revised: =

Original: ( X A , D A ) and B = ( X B , D B ) .
Revised: ( X A , D A ) and B = ( X B , D B ) .

Original: Here, X A , X B ⊂ R M and D A , D B are distributions over them (resp.).
Revised: Here, X A , X B ⊂ R M and D A , D B are distributions over them (resp.).

Original: The algorithm is provided with two independent datasets S A
Revised: The algorithm is provided with two independent datasets S A

Original: = { a i } m 

Revised: = { a i } m

Original: i =1 and S B = { b j } m 2 j =1 of samples from the two domains that were sampled in the following manner:
Revised: i =1 and S B = { b j } m 2 j =1 of samples from the two domains that were sampled in the following manner:

Original: [Equation]
Revised: [Equation]

Original: We denote, D A,B := D
Revised: We denote, D A,B := D

Original: A × D B the distribution of sampling ( a , b ) for a ∼ D A and b ∼ D B independently.
Revised: A × D B the distribution of sampling ( a , b ) for a ∼ D A and b ∼ D B independently.

Original: We assume a generative model, in which b is speciﬁed by a sample a and a speciﬁcation c from a third unknown domain C =
Revised: We assume a generative model, in which b is speciﬁed by a sample a and a speciﬁcation c from a third unknown domain C =

Original: ( X C , D C ) of speciﬁcations where D C is a distribution over the metric space X C ⊂ R N .
Revised: ( X C , D C ) of speciﬁcations where D C is a distribution over the metric space X C ⊂ R N .

Original: Formally, there is an invertible function u ( b ) =
Revised: Formally, there is an invertible function u ( b ) =

Original: ( u 1 ( b ) , u 2 ( b ))
Revised: ( u 1 ( b ) , u 2 ( b ))

Original: ∈
Revised: ∈

Original: X A × X C that takes a sample b ∈ X B and returns the content u 1 ( b ) of b and the speciﬁcation u 2 ( b ) of b .
Revised: X A × X C that takes a sample b ∈ X B and returns the content u 1 ( b ) of b and the speciﬁcation u 2 ( b ) of b .

Original: The goal is to learn a target function y :
Revised: The goal is to learn a target function y :

Original: X A × X B → X B such that: y ( a , b ) ∼ D B where: a ∼ D A , b ∼ D B , a | = b and u ( y ( a , b ))
Revised: X A × X B → X B such that: [Equation] [Equation]

Original: = ( a , u 2 ( b ))
Revised: ∼

Original: [Equation]
Revised: [Equation]

Original: Informally, the function y takes two samples a and b and returns the analog of a in B that has the speciﬁcation of b .
Revised: Informally, the function y takes two samples a and b and returns the analog of a in B that has the speciﬁcation of b .

Original: For example, A is the domain of images of persons, B is the domain of images of persons with sunglasses and C is the domain of images of sunglasses.
Revised: For example, A is the domain of images of persons, B is the domain of images of persons with sunglasses and C is the domain of images of sunglasses.

Original: The function y takes an image of a person and an image of a person with sunglasses and returns an image of the ﬁrst person with the speciﬁed sunglasses.
Revised: The function y takes an image of a person and an image of a person with sunglasses and returns an image of the ﬁrst person with the speciﬁed sunglasses.

Original: For simplicity, we assume that the target function is extended to inputs ( b 1 , b 2 ) ∈
Revised: For simplicity, we assume that the target function is extended to inputs ( b 1 , b 2 ) ∈

Original: X 2 B and u ( y ( b 1 , b 2 ))
Revised: X 2 B and u ( y ( b 1 , b 2 ))

Original: = ( u 1 ( b 1 ) , u 2 ( b 2 )) .
Revised: = ( u 1 ( b 1 ) , u 2 ( b 2 )) .

Original: In other words, b 1 and b 2 are mapped to a third b that has the content of b 1 and the speciﬁcation of b 2 .
Revised: In other words, b 1 and b 2 are mapped to a third b that has the content of b 1 and the speciﬁcation of b 2 .

Original: In particular, u ( y ( b , b )) =
Revised: In particular, u ( y ( b , b )) =

Original: ( u 1 ( b ) , u 2 ( b ))
Revised: ( u 1 ( b ) , u 2 ( b ))

Original: =
Revised: =

Original: u ( b ) and, therefore, y ( b , b )
Revised: u ( b ) and, therefore, y ( b , b )

Original: = b .
Revised: = b .

Original: Note that within Eq. 2, there is an assumption on the underlying distributions D A and D B .
Revised: Note that within Eq. 2, there is an assumption on the underlying distributions D A and D B .

Original: Using the concrete example, our framework assumes that the distribution of images of persons with sunglasses andthat the distribution of images of persons without them is the same, except for the sunglasses.
Revised: Using the concrete example, our framework assumes that the distribution of images of persons with sunglasses and the distribution of images of persons without them is the same, except for the sunglasses.

Original: Otherwise, the distribution of the samples generated by y when resampling a would not be the same as D B .
Revised: Otherwise, the distribution of the samples generated by y when resampling a would not be the same as D B .

Original: Note that we do not enforce this assumption on the data, and only employ it for our theoretical results, to avoid additional terms.
Revised: Note that we do not enforce this assumption on the data, and only employ it for our theoretical results, to avoid additional terms.

Original: For two functions f 1 , f 2 : X → R and a distribution D over X , we deﬁne the generalization risk between f 1 and f 2 as follows:
Revised: For two functions f 1 , f 2 : X → R and a distribution D over X , we deﬁne the generalization risk between f 1 and f 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: For a loss function (cid:96) : R M × R M → [0 , ∞ ) .
Revised: For a loss function (cid:96) : R M × R M → [0 , ∞ ) .

Original: Typically, we use the L 1 ( u , v ) :
Revised: Typically, we use the L 1 ( u , v ) :

Original: = (cid:107) u − v (cid:107) 1 or
Revised: = (cid:107) u − v (cid:107) 1 or

Original: L 2 ( u , v ) :
Revised: L 2 ( u , v ) :

Original: = (cid:107) u − v (cid:107) 22 losses.
Revised: = (cid:107) u − v (cid:107) 22 losses.

Original: The goal of the algorithm is to return a hypothesis h ∈ H , such that h [Equation] R M , that minimizes the generalization risk,
Revised: The goal of the algorithm is to return a hypothesis h ∈ H , such that h : R M × R M → R M , that minimizes the generalization risk,

Original: [Equation]
Revised: [Equation]

Original: This quantity measures the expected loss of h in mapping two samples a ∼ D A and b ∼ D B to the analog y ( a , b ) of a , that has the speciﬁcation u 2 ( b ) of b .
Revised: This quantity measures the expected loss of h in mapping two samples a ∼ D A and b ∼ D B to the analog y ( a , b ) of a , that has the speciﬁcation u 2 ( b ) of b .

Original: The main challenge is that the algorithm does not observes paired examples of the form (( a , b ) , y ( a , b )) as a direct supervision for learning the mapping y :
Revised: The main challenge is that the algorithm does not observes paired examples of the form (( a , b ) , y ( a , b )) as a direct supervision for learning the mapping y :

Original: X A × X B → X B .
Revised: X A × X B → X B .

Original:  M ETHOD
Revised:  M ETHOD

Original: In order to learn the mapping y , we only use an encoder-decoder architecture, in which the encoder receives two input samples and the decoder produces a single output sample that borrows from both input samples.
Revised: In order to learn the mapping y , we only use an encoder-decoder architecture, in which the encoder receives two input samples and the decoder produces a single output sample that borrows from both input samples.

Original: As we discuss in Sec. 2, the goal of the algorithm is to learn a mapping h = g ◦ f ∈ H such that: g ◦ f ( a , b ) ≈
Revised: As we discuss in Sec. 2, the goal of the algorithm is to learn a mapping h = g ◦ f ∈ H such that: g ◦ f ( a , b ) ≈

Original: y
Revised: y

Original: ( a , b ) .
Revised: ( a , b ) .

Original: Here, f serves as an encoder and g as a decoder.
Revised: Here, f serves as an encoder and g as a decoder.

Original: The encoder f in our framework is a member of a set of encoders F , each decomposable into two parts and takes the following form:
Revised: The encoder f in our framework is a member of a set of encoders F , each decomposable into two parts and takes the following form:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: e 1 ( a ) ∼ e 1 ◦ D A e 1 ( b ) ∼ e 1 ◦ D B a ∼ D A b ∼ D B e 2 ( b ) f ( a , a ) f ( b , b ) disc ee eg g where

Original: where e 1 : R M → R E 1 serves as an encoder of shared content and e 2 : R M → R E 2 serves as an encoder of speciﬁc content.
Revised: e 1 : R M → R E 1 serves as an encoder of shared content and e 2 : R M → R E 2 serves as an encoder of speciﬁc content.

Original: Here, E 1 and E 2 are the dimensions of the encodings.
Revised: Here, E 1 and E 2 are the dimensions of the encodings.

Original: The decoder, g is a member of a set of decoders M .
Revised: The decoder, g is a member of a set of decoders M .

Original: Each member of M is a function g : R E 1 + E 2 → R M .
Revised: Each member of M is a function g : R E 1 + E 2 → R M .

Original: In order to learn the functions f
Revised: In order to learn the functions f

Original: and g , we apply the following min-max optimization:
Revised: and g , we apply the following min-max optimization:

Original: [Equation]
Revised: [Equation]

Original: e 1 ( a ) ∼ e 1 ◦ D A e 1 ( b ) ∼ e 1 ◦ D B a ∼ D

Original: A b ∼ D B e 2 ( b )

Original: f ( a , a ) f ( b , b ) disc ee eg g for some weight parameter λ > 0 , of the following training losses (see Fig. 1): 
Revised: (6) for some weight parameter λ > 0 , of the following training losses (see Fig. 1): [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: where 0 E 2 is the vector of zeros of length E 2 , d is a discriminator network, and l ( p, q ) =
Revised: where 0 E 2 is the vector of zeros of length E 2 , d is a discriminator network, and l ( p, q ) =

Original: − ( q log( p ) + (1 − q ) log(1 − p )) is the binary cross entropy loss for p ∈ [0 , 1] and q ∈ { 0 , 1 } .
Revised: − ( q log( p ) + (1 − q ) log(1 − p )) is the binary cross entropy loss for p ∈ [0 , 1] and q ∈ { 0 , 1 } .

Original: The discriminator d is a member of a set of discriminators C that locates functions d : R M → [0 , 1] .
Revised: The discriminator d is a member of a set of discriminators C that locates functions d : R M → [0 , 1] .

Original: The discriminator d is trained to minimize L D and Eq.
Revised: The discriminator d is trained to minimize L D and Eq.

Original: 9 is a domain confusion term (Ganin et al., 2016) encouraging the distribution e 1 ◦ D A to be similar to the distribution e 1 ◦ D B .
Revised: 9 is a domain confusion term (Ganin et al., 2016) encouraging the distribution e 1 ◦ D A to be similar to the distribution e 1 ◦ D B .

Original: Here and elsewhere, the composition f ◦ D of a function f and a distribution D denotes the distribution of f ( x ) for x ∼ D .
Revised: Here and elsewhere, the composition f ◦ D of a function f and a distribution D denotes the distribution of f ( x ) for x ∼ D .

Original:  A NALYSIS
Revised:  A NALYSIS

Original: In this section, we provide a theoretical analysis for the success of the proposed method.
Revised: In this section, we provide a theoretical analysis for the success of the proposed method.

Original: For this purpose, we recall a few technical notations (Cover & Thomas, 2006): the expectation and probability operators symbols E , P , the Shannon entropy (discrete or continuous) H ( X ) :
Revised: For this purpose, we recall a few technical notations (Cover & Thomas, 2006): the expectation and probability operators symbols E , P , the Shannon entropy (discrete or continuous) H ( X ) :

Original: = − E X [log P
Revised: = − E X [log P

Original: [ X ]] , the conditional entropy H ( X | Y ) :
Revised: [ X ]] , the conditional entropy H ( X | Y ) :

Original: = H ( X, Y ) − H ( Y ) , the (conditional) mutual information (discrete or continuous)
Revised: = H ( X, Y ) − H ( Y ) , the (conditional) mutual information (discrete or continuous)

Original: I
Revised: I

Original: ( X ; Y | Z ) :
Revised: ( X ; Y | Z ) :

Original: = H ( X | Z ) − H ( X | Y, Z ) , the Kullback-Leibler (KL) divergence
Revised: = H ( X | Z ) − H ( X | Y, Z ) , the Kullback-Leibler (KL) divergence

Original: D KL ( p (cid:107) q ) :
Revised: D KL ( p (cid:107) q ) :

Original: =
Revised: =

Original: E x ∼ p
Revised: E x ∼ p

Original: [log( p ( x ) /q
Revised: [log( p ( x ) /q

Original: ( x ))] , and the total correlation TC ( z ) :
Revised: ( x ))] , and the total correlation TC ( z ) :

Original: = D KL ( P [ z ] (cid:107) (cid:81)
Revised: = D KL ( P [ z ] (cid:107) (cid:81)

Original: i P [ z i ]) , where P [ z i ] is the marginal distribution of the i ’th component of z .
Revised: i P [ z i ]) , where P [ z i ] is the marginal distribution of the i ’th component of z .

Original: In particular, TC ( z ) is zero if and only if the components of z are independent, in which case we say that z is disentangled.
Revised: In particular, TC ( z ) is zero if and only if the components of z are independent, in which case we say that z is disentangled.

Original: For two distributions D 1 and D 2 , we deﬁne the C -discrepancy between them to be disc C
Revised: For two distributions D 1 and D 2 , we deﬁne the C -discrepancy between them to be disc C

Original: ( D 1 , D 2 ) :
Revised: ( D 1 , D 2 ) :

Original: = sup c 1 ,c [Equation] | R [Equation]
Revised: = sup c 1 ,c 2 ∈C | R D 1 [ c 1 , c 2 ]

Original: 1 [ c 1 , c 2 ] − R

Original: [Equation]
Revised: [Equation]

Original: 2 [ c

Original: [Equation]
Revised: [Equation]

Original: , c 2 ] |= sup [Equation] [Equation] |
Revised: ] |

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  )) −
Revised: x )) | .

Original: [Equation]

Original: ( c 1 ( x ) , c 2 ( x )) | .

Original: The discrepancy behaves as an adversarial distance measure between two distributions, where d ( x ) =
Revised: The discrepancy behaves as an adversarial distance measure between two distributions, where d ( x ) =

Original: (cid:96) ( c 1 ( x ) , c 2 ( x )) is the discriminator that tries to differentiate between D 1 and D 2 , for c 1 , c 2 ∈ C .
Revised: (cid:96) ( c 1 ( x ) , c 2 ( x )) is the discriminator that tries to differentiate between D 1 and D 2 , for c 1 , c 2 ∈ C .

Original: This quantity is being employed in Chazelle (2000); Ben-david et al.
Revised: This quantity is being employed in Chazelle (2000); Ben-david et al.

Original: ; Mansour et al.
Revised: ; Mansour et al.

Original: ; Cortes & Mohri (2014).
Revised: ; Cortes & Mohri (2014).

Original: 4. G ENERALIZATION B OUND
Revised: 4. G ENERALIZATION B OUND

Original: Thm.
Revised: Thm.

Original: 1 upper bounds the generalization risk, based on terms that can be minimized during training, as well as on approximation terms.
Revised: 1 upper bounds the generalization risk, based on terms that can be minimized during training, as well as on approximation terms.

Original: It is similar in fashion to the classic domain adaptation bounds proposed by Ben-David et al.
Revised: It is similar in fashion to the classic domain adaptation bounds proposed by Ben-David et al.

Original: ; Mansour et al.
Revised: ; Mansour et al.

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume that the loss function (cid:96) is symmetric and obeys the triangle inequality.
Revised: Assume that the loss function (cid:96) is symmetric and obeys the triangle inequality.

Original: Then, for any autoencoder h = g ◦ f ∈ H , such that f ( x 1 , x 2 )
Revised: Then, for any autoencoder h = g ◦ f ∈ H , such that f ( x 1 , x 2 )

Original: = ( e 1 ( x 1 ) , e 2 ( x 2 ))
Revised: = ( e 1 ( x 1 ) , e 2 ( x 2 ))

Original: ∈ F is an encoder and g ∈ M is a decoder, the following holds,
Revised: ∈ F is an encoder and g ∈ M is a decoder, the following holds,

Original: [Equation]
Revised: [Equation]

Original: where D B, ˆ B is the distribution of ( b , b ) where b ∼ D B .
Revised: where D B, ˆ B is the distribution of ( b , b ) where b ∼ D B .

Original: (The proofs can be found in the appendix.)
Revised: (The proofs can be found in the appendix.)

Original: Thm. 1 provides an upper bound on the generalization risk R D A,B
Revised: Thm. 1 provides an upper bound on the generalization risk R D A,B

Original: [ h, y ] , which is the argument that we would like to minimize.
Revised: [ h, y ] , which is the argument that we would like to minimize.

Original: The upper bound is decomposed of three terms: a reconstruction error, an approximation error and a discrepancy term.
Revised: The upper bound is decomposed of three terms: a reconstruction error, an approximation error and a discrepancy term.

Original: The ﬁrst term,
Revised: The ﬁrst term,

Original: [Equation]
Revised: [Equation]

Original:  is the reconstruction error for samples b ∼
Revised: 11) is the reconstruction error for samples b ∼

Original: D B .
Revised: D B .

Original: Since we do not have full access to D B , we minimize its empirical version (see Eq.
Revised: Since we do not have full access to D B , we minimize its empirical version (see Eq.

Original: The second term,min g ∈M (cid:110) R D A,B [ g ◦ f, y ] + R D
Revised: The second term,

Original: [Equation]
Revised: [Equation]

Original: , ˆ B [ g ◦ f, y ] (cid:111) , measures the minimal error obtained by a best ﬁtting g ∈ M , such that g ◦ f ≈ y for inputs ( a , b ) ∼ D A,B and for inputs ( b , b ) ∼ D B, ˆ B .
Revised: g ◦ f, y ] + R D B, ˆ B [ g ◦ f, y ] (cid:111) , measures the minimal error obtained by a best ﬁtting g ∈ M , such that g ◦ f ≈ y for inputs ( a , b ) ∼ D A,B and for inputs ( b , b ) ∼ D B, ˆ B .

Original: Similar to (Ben-David et al., 2010; Mansour et al., 2009), this term is assumed to be small and is decreased as
Revised: Similar to (Ben-David et al., 2010; Mansour et al., 2009), this term is assumed to be small and is decreased as

Original: M ’s capacity is increased.
Revised: M ’s capacity is increased.

Original: The third term, disc M ( f ◦ D A,B , f ◦ D B, ˆ B ) , is the discrepancy between the distributions f ◦ D A,B and f ◦ D B, ˆ B .
Revised: The third term, disc M ( f ◦ D A,B , f ◦ D B, ˆ B ) , is the discrepancy between the distributions f ◦ D A,B and f ◦ D B, ˆ B .

Original: This term is small, if the distributions of ( e 1 ( a ) , e 2 ( b ))
Revised: This term is small, if the distributions of ( e 1 ( a ) , e 2 ( b ))

Original: (for a ∼ D A and b ∼ D B independently) and ( e 1 ( b ) , e 2 ( b )) (for b ∼ D B ) are close to each other.
Revised: (for a ∼ D A and b ∼ D B independently) and ( e 1 ( b ) , e 2 ( b )) (for b ∼ D B ) are close to each other.

Original: Since e 1 ( a ) and e 2 ( b ) are independent of each other (from the factorization D A,B
Revised: Since e 1 ( a ) and e 2 ( b ) are independent of each other (from the factorization D A,B

Original: = D
Revised: = D

Original: A × D B ), if this term is small, then, e 1 ( b ) and e 2 ( b ) weakly depend on each other.
Revised: A × D B ), if this term is small, then, e 1 ( b ) and e 2 ( b ) weakly depend on each other.

Original: Moreover, if the discrepancy term is zero, then, e 1 ( b ) and e 2 ( b ) are independent of each other.
Revised: Moreover, if the discrepancy term is zero, then, e 1 ( b ) and e 2 ( b ) are independent of each other.

Original: While one can minimize the discrepancy term explicitly, by minimizing it with respect to e 1 and e 2 , using a discriminator, we found empirically that this confusion term, which involves both parts of the embedding, is highly unstable.
Revised: While one can minimize the discrepancy term explicitly, by minimizing it with respect to e 1 and e 2 , using a discriminator, we found empirically that this confusion term, which involves both parts of the embedding, is highly unstable.

Original: Instead, we show theoretically and empirically that there is a high likelihood for a disentangled representation (where e 1 ( b ) and e 2 ( b ) are independent) to emerge, and the discrepancy term can be replaced with the following discrepancy disc M (cid:48)
Revised: Instead, we show theoretically and empirically that there is a high likelihood for a disentangled representation (where e 1 ( b ) and e 2 ( b ) are independent) to emerge, and the discrepancy term can be replaced with the following discrepancy disc M (cid:48)

Original: ( e 1 ◦ D A , e 1 ◦ D B ) , which measures the closeness between the distributions of e 1 ( a ) and of e 1 ( b ) for a ∼ D A and b ∼ D B , as is done in Eq.
Revised: ( e 1 ◦ D A , e 1 ◦ D B ) , which measures the closeness between the distributions of e 1 ( a ) and of e 1 ( b ) for a ∼ D A and b ∼ D B , as is done in Eq.

Original: Here, M (cid:48) is a set of discriminators that are similar in complexity to the ones in M .
Revised: Here, M (cid:48) is a set of discriminators that are similar in complexity to the ones in M .

Original: This discrepancy is simpler than the one in Eq.
Revised: This discrepancy is simpler than the one in Eq.

Original: 10, since it does not involve a comparison of e 2 between two distributions, nor the interaction between e 1 and e 2 .
Revised: 10, since it does not involve a comparison of e 2 between two distributions, nor the interaction between e 1 and e 2 .

Original: In Lem.
Revised: In Lem.

Original: 1, we show that if e 1 ( b ) and e 2 ( b ) are independent, then, disc ( f ◦ D A,B , f ◦ D B, ˆ B ) ≤ disc ( e 1 ◦ D A , e 1 ◦ D B ) .
Revised: 1, we show that if e 1 ( b ) and e 2 ( b ) are independent, then, disc ( f ◦ D A,B , f ◦ D B, ˆ B ) ≤ disc ( e 1 ◦ D A , e 1 ◦ D B ) .

Original: Therefore, if a disentangled representation occurs, we can minimize disc ( f ◦
Revised: Therefore, if a disentangled representation occurs, we can minimize disc ( f ◦

Original: D A,B , f ◦ D B, ˆ B ) , by minimizing disc ( e 1 ◦ D A , e 1 ◦ D B ) instead.
Revised: D A,B , f ◦ D B, ˆ B ) , by minimizing disc ( e 1 ◦ D A , e 1 ◦ D B ) instead.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let M be the set of neural networks of the form: c ( x ) =
Revised: Let M be the set of neural networks of the form: c ( x ) =

Original: φ ( W r . . .
Revised: φ ( W r . . .

Original: φ ( W 2 φ ( W 1 x + q ))) , where, W i ∈ R d i × d i +1 for i ∈ { 1 , . . .
Revised: φ ( W 2 φ ( W 1 x + q ))) , where, W i ∈ R d i × d i +1 for i ∈ { 1 , . . .

Original: , r − 1 } , q ∈ R d 2 and d 1 = E 1 + E 2 .
Revised: , r − 1 } , q ∈ R d 2 and d 1 = E 1 + E 2 .

Original: In addition, φ ( x 1 , . . .
Revised: In addition, φ ( x 1 , . . .

Original: , x k ) =
Revised: , x k ) =

Original: ( φ 1 ( x 1 ) , . . .
Revised: ( φ 1 ( x 1 ) , . . .

Original: , φ 1 ( x k )) , for k ∈ N , ( x 1 , . . .
Revised: , φ 1 ( x k )) , for k ∈ N , ( x 1 , . . .

Original: , x k ) ∈ R k and a non-linear activation function φ 1 : R → R .
Revised: , x k ) ∈ R k and a non-linear activation function φ 1 : R → R .

Original: Let M (cid:48) be the same as M with d 1 = E 1 (instead of d 1 = E 1 + E 2 ).
Revised: Let M (cid:48) be the same as M with d 1 = E 1 (instead of d 1 = E 1 + E 2 ).

Original: Let f ( x )
Revised: Let f ( x )

Original: =
Revised: =

Original: ( e 1 ( x ) , e 2 ( x )) be an encoder and assume that: e 1 ( b ) | =
Revised: ( e 1 ( x ) , e 2 ( x )) be an encoder and assume that: e 1 ( b ) | =

Original: e 2 ( b ) .
Revised: e 2 ( b ) .

Original: Then,
Revised: Then,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 4. E MERGENCE OF D ISENTANGLED R EPRESENTATIONS
Revised: 4. E MERGENCE OF D ISENTANGLED R EPRESENTATIONS

Original: The following results are very technical and inherit many of the assumptions used by previous work.
Revised: The following results are very technical and inherit many of the assumptions used by previous work.

Original: We therefore state the results informally here and leave the complete exposition to the appendix.
Revised: We therefore state the results informally here and leave the complete exposition to the appendix.

Original: First, we extend Proposition 5.2 of Achille & Soatto (2018) from the case of multiclass classiﬁcation to the case of autoencoders.
Revised: First, we extend Proposition 5.2 of Achille & Soatto (2018) from the case of multiclass classiﬁcation to the case of autoencoders.

Original: In their work, the aim is to show the conditions in which a mid-level representation f of a multi-class classiﬁcation neural network
Revised: In their work, the aim is to show the conditions in which a mid-level representation f of a multi-class classiﬁcation neural network

Original: h
Revised: h

Original: =
Revised: =

Original: c ◦ f is both disentangled, i.e.,
Revised: c ◦ f is both disentangled, i.e.,

Original: TC ( f ( b )) is small, and is minimal, i.e., I ( f ( b ); b ) is small, where b is an input random variable.
Revised: TC ( f ( b )) is small, and is minimal, i.e., I ( f ( b ); b ) is small, where b is an input random variable.

Original: In their Proposition 5.2, they focus on a linear representation, i.e., f = W is a linear transformation, and they introduce a tight upper bound on the sum
Revised: In their Proposition 5.2, they focus on a linear representation, i.e., f = W is a linear transformation, and they introduce a tight upper bound on the sum

Original: TC ( W b )
Revised: TC ( W b )

Original: + I ( W b ; b ) .
Revised: + I ( W b ; b ) .

Original: In the general case, their goal is to show that for a neural network
Revised: In the general case, their goal is to show that for a neural network

Original: h =
Revised: h =

Original: c ◦ f , both quantities TC ( f ( b )) and I ( f ( b ); b ) are small, when f that is a high level representation of the input.
Revised: c ◦ f , both quantities TC ( f ( b )) and I ( f ( b ); b ) are small, when f that is a high level representation of the input.

Original: Unfortunately, they were unable to show that both terms are small simultaneously.
Revised: Unfortunately, they were unable to show that both terms are small simultaneously.

Original: Therefore, in their Cor. 5.3, they extend the bound of their Proposition 5.2 to show that only the mutual information I ( f ( b ); b ) is small and assume that the components of each mid-level representation of b in the layers of f are uncorrelated, which is a very restrictive assumption.
Revised: Therefore, in their Cor. 5.3, they extend the bound of their Proposition 5.2 to show that only the mutual information I ( f ( b ); b ) is small and assume that the components of each mid-level representation of b in the layers of f are uncorrelated, which is a very restrictive assumption.

Original: In our Lem. 2, we provide an upper bound for TC ( f ( b )) that is similar in fashion to their bounds.
Revised: In our Lem. 2, we provide an upper bound for TC ( f ( b )) that is similar in fashion to their bounds.

Original: The main differentiating factor is that we deal with an autoencoder h = g ◦ f .
Revised: The main differentiating factor is that we deal with an autoencoder h = g ◦ f .

Original: In this case, the mutual information I ( h ( b ); b ) is expected to be large, since h ( b ) is trained to recover b and by the dataprocessing inequality, I ( f ( b ); b ) ≥ I ( h ( b ); b ) and therefore, I ( f ( b ); b ) is also large in this case.
Revised: In this case, the mutual information I ( h ( b ); b ) is expected to be large, since h ( b ) is trained to recover b and by the dataprocessing inequality, I ( f ( b ); b ) ≥ I ( h ( b ); b ) and therefore, I ( f ( b ); b ) is also large in this case.

Original: As a result, no upper bound is given on the mutual information term I ( f ( b ); b ) .
Revised: As a result, no upper bound is given on the mutual information term I ( f ( b ); b ) .

Original: This is unlike the information bottleneck principle, which guides the classiﬁcation setting of Achille & Soatto (2018).
Revised: This is unlike the information bottleneck principle, which guides the classiﬁcation setting of Achille & Soatto (2018).

Original: Our upper bound on TC ( f ( b ))
Revised: Our upper bound on TC ( f ( b ))

Original: includes the term − I ( h ( b ); b ) , and consequently,
Revised: includes the term − I ( h ( b ); b ) , and consequently,

Original: TC ( f ( b )) tends to be smaller (increased disentanglement) as I ( h ( b ); b ) increases.
Revised: TC ( f ( b )) tends to be smaller (increased disentanglement) as I ( h ( b ); b ) increases.

Original: Additionally, the upper bound sums a term d 1 · q ( α ) .
Revised: Additionally, the upper bound sums a term d 1 · q ( α ) .

Original: Here, d 1 is the dimension of f ( b ) , α denotes the amount of regularization in the weights of f and q ( α ) is monotonically decreasing as α tends to zero.
Revised: Here, d 1 is the dimension of f ( b ) , α denotes the amount of regularization in the weights of f and q ( α ) is monotonically increasing as α tends to zero.

Original: Therefore, the disentanglement tends to be larger for an autoencoder h
Revised: Therefore, the disentanglement tends to be larger for an autoencoder h

Original: = g ◦ f , such that f is regularized and the mutual information
Revised: = g ◦ f , such that f is regularized and the mutual information

Original: I ( h ( b ); b ) is large.
Revised: I ( h ( b ); b ) is large.

Original: In our analysis, we do not require f to be a linear transformation and we do not assume that the components of each mid-level representation of b in the layers of f are uncorrelated.
Revised: In our analysis, we do not require f to be a linear transformation and we do not assume that the components of each mid-level representation of b in the layers of f are uncorrelated.

Original: Lemma 2 (Informal) .
Revised: Lemma 2 (Informal) .

Original: Let b ∼ D B be a distribution and h =
Revised: Let b ∼ D B be a distribution and h =

Original: g ◦ f an autoencoder.
Revised: g ◦ f an autoencoder.

Original: Let d 1 be the dimension of f ( b ) and d 2 the dimension of the layer previous to f ( b ) .
Revised: Let d 1 be the dimension of f ( b ) and d 2 the dimension of the layer previous to f ( b ) .

Original: Under some assumptions on the weights of the encoder, there is a monotonically decreasing function q ( α ) for α > 0 such that:
Revised: Under some assumptions on the weights of the encoder, there is a monotonically decreasing function q ( α ) for α > 0 such that:

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 13 bounds the total correlation of f ( b ) , which measures the amount of dependence between the components of the encoder on samples in B .
Revised: 13 bounds the total correlation of f ( b ) , which measures the amount of dependence between the components of the encoder on samples in B .

Original: The bounds has three terms: d 1 · q ( α ) , − I ( h ( b ) , b ) and O ( d 1 /d 2 ) .
Revised: The bounds has three terms: d 1 · q ( α ) , − I ( h ( b ) , b ) and O ( d 1 /d 2 ) .

Original: In this formulation, α denotes the amount of regularization in the weights of f .
Revised: In this formulation, α denotes the amount of regularization in the weights of f .

Original: In addition, q ( α ) is monotonically increasing as α tends to zero.
Revised: In addition, q ( α ) is monotonically increasing as α tends to zero.

Original: The term I ( h ( b ); b ) measures the mutual information between the input b and output h ( b ) of the autoencoder h .
Revised: The term I ( h ( b ); b ) measures the mutual information between the input b and output h ( b ) of the autoencoder h .

Original: Since the mutual information is subtracted in the right hand side, the larger it is, the smaller TC ( f ( b )) should be.
Revised: Since the mutual information is subtracted in the right hand side, the larger it is, the smaller TC ( f ( b )) should be.

Original: The last term, O ( d 1 /d 2 ) measures the ratio between the dimension of the output of f and the dimension of the previous layer of f .
Revised: The last term, O ( d 1 /d 2 ) measures the ratio between the dimension of the output of f and the dimension of the previous layer of f .

Original: Thus, this quantity is small whenever there is a signiﬁcance reduction in the dimension in the application of the last layer of f .
Revised: Thus, this quantity is small whenever there is a signiﬁcance reduction in the dimension in the application of the last layer of f .

Original: Therefore, there is a tradeoff between the amount of regularization in the weights of f and the mutual information
Revised: Therefore, there is a tradeoff between the amount of regularization in the weights of f and the mutual information

Original: I ( h ( b ); b ) .
Revised: I ( h ( b ); b ) .

Original: If there is small regularization, then, the autoencoder is able to produce better reconstruction h ( b ) ≈ b , and therefore, a larger value of I ( h ( b ); b ) .
Revised: If there is small regularization, then, the autoencoder is able to produce better reconstruction h ( b ) ≈ b , and therefore, a larger value of I ( h ( b ); b ) .

Original: On the other hand, small regularization leads to a higher value of q ( α ) .
Revised: On the other hand, small regularization leads to a higher value of q ( α ) .

Original: The bound relies on the mutual information between the inputs and outputs of the autoencoder to be large.
Revised: The bound relies on the mutual information between the inputs and outputs of the autoencoder to be large.

Original: The following lemma provides an argument why this is the case when the expected reconstruction error of the autoencoder is small.
Revised: The following lemma provides an argument why this is the case when the expected reconstruction error of the autoencoder is small.

Original: Lemma 3 (Informal) .
Revised: Lemma 3 (Informal) .

Original: Let b ∼ D B be a distribution over a discrete set X B and h =
Revised: Let b ∼ D B be a distribution over a discrete set X B and h =

Original: g ◦ f an autoencoder.
Revised: g ◦ f an autoencoder.

Original: Assume that ∀ x 1 (cid:54) = x 2 ∈ X B : (cid:107) x 1 − x 2 (cid:107) 1 > ∆ .
Revised: Assume that ∀ x 1 (cid:54) = x 2 ∈ X B : (cid:107) x 1 − x 2 (cid:107) 1 > ∆ .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: The above lemma asserts that if the samples in D B are well separated, whenever the autoencoder has a small expected reconstruction error, E [ (cid:107) h ( b ) − b (cid:107) 1 ] , then, the mutual information I ( h ( b ); b ) is at least a large portion of H ( b ) .
Revised: The above lemma asserts that if the samples in D B are well separated, whenever the autoencoder has a small expected reconstruction error, E [ (cid:107) h ( b ) − b (cid:107) 1 ] , then, the mutual information I ( h ( b ); b ) is at least a large portion of H ( b ) .

Original: Therefore, we conclude that if the autoencoder generalizes well, then, it also maximizes the mutual information I ( h ( b ); b ) .
Revised: Therefore, we conclude that if the autoencoder generalizes well, then, it also maximizes the mutual information I ( h ( b ); b ) .

Revised: [Table]

Original: To conclude the analysis: for a small enough reconstruction error, when training the autoencoder, the mutual information between the autoencoder’s input and output is high (Lem. 10), which implies that the individual coordinates of the representation layer are almost independent of each other (Lem. 9).
Revised: To conclude the analysis: for a small enough reconstruction error, when training the autoencoder, the mutual information between the autoencoder’s input and output is high (Lem. 10), which implies that the individual coordinates of the representation layer are almost independent of each other (Lem. 9).

Original: When using part of the representation to encode the information that exists in domain A (the shared[Table] part), the other part would contain coordinates that are weakly dependent of the features encoded in
Revised: When using part of the representation to encode the information that exists in domain A (the shared part), the other part would contain coordinates that are weakly dependent of the features encoded in

Original: A .
Revised: A .

Original: In such a case, we can train with a GAN that involves only the shared representation (Lem. 1).
Revised: In such a case, we can train with a GAN that involves only the shared representation (Lem. 1).

Original: That way, we can upper bound the generalization error expressed in Thm. 1, using relatively simple loss terms, as is done in Sec. 3.
Revised: That way, we can upper bound the generalization error expressed in Thm. 1, using relatively simple loss terms, as is done in Sec. 3.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our method on three additive facial attributes: eyewear, facial hair, and smile.
Revised: We evaluate our method on three additive facial attributes: eyewear, facial hair, and smile.

Original: Images from the celebA face image dataset by Yang et al.
Revised: Images from the celebA face image dataset by Yang et al.

Original: (2015) were used, since these are conveniently annotated as having the attribute or not.
Revised: (2015) were used, since these are conveniently annotated as having the attribute or not.

Original: The images without the attribute (no glasses, or no facialhair, or no smile) were used as domain A in each of the experiments.
Revised: The images without the attribute (no glasses, or no facialhair, or no smile) were used as domain A in each of the experiments.

Original: Note that three different A domains were used.
Revised: Note that three different A domains were used.

Original: As the second domain B , we used the images labeled as having glasses, having facial hair, or smiling, according to the experiment.
Revised: As the second domain B , we used the images labeled as having glasses, having facial hair, or smiling, according to the experiment.

Original: Our underlying network architecture adapts the architecture used by Lample et al.
Revised: Our underlying network architecture adapts the architecture used by Lample et al.

Original: (2017), which is based on (Isola et al., 2017a), where we use Instance Normalization (Ulyanov et al., 2016) instead of
Revised: (2017), which is based on (Isola et al., 2017a), where we use Instance Normalization (Ulyanov et al., 2016) instead of

Original: Batch Normalization (Ioffe & Szegedy, 2015), and without dropout.
Revised: Batch Normalization (Ioffe & Szegedy, 2015), and without dropout.

Original: Let C k denote a ConvolutionInstanceNorm-ReLU layer with k ﬁlters, where a kernel size of 4 × 4 , with a stride of 2, and a padding of 1 is used.
Revised: Let C k denote a ConvolutionInstanceNorm-ReLU layer with k ﬁlters, where a kernel size of 4 × 4 , with a stride of 2, and a padding of 1 is used.

Original: The activations of the encoders e 1 , e 2 are leaky-ReLUs with a slope of 0.2 and the deocder g employs ReLUs.
Revised: The activations of the encoders e 1 , e 2 are leaky-ReLUs with a slope of 0.2 and the deocder g employs ReLUs.

Original: e 1 has the following layers C 32 , C 64 , C 128 , C 256 , C 512 , C 512 − d ; ehas a slightly lower capacity C 32 , C 64 , C 128 , C 128 , C 128 , C d , where d = 25 .
Revised: e 1 has the following layers C 32 , C 64 , C 128 , C 256 , C 512 , C 512 − d ; ehas a slightly lower capacity C 32 , C 64 , C 128 , C 128 , C 128 , C d , where d = 25 .

Original: The input images have a size of 128 × 128 , and the encoding is of size 512 × 2 × 2 (split between the e 1 and e 2 ).
Revised: The input images have a size of 128 × 128 , and the encoding is of size 512 × 2 × 2 (split between the e 1 and e 2 ).

Original: g is symmetric to the encoders and employs transposed convolutions for the upsampling.
Revised: g is symmetric to the encoders and employs transposed convolutions for the upsampling.

Original: In the ﬁrst set of experiments, we add the relevant content from a random image b ∈ B into an image from a .
Revised: In the ﬁrst set of experiments, we add the relevant content from a random image b ∈ B into an image from a .

Original: The results are given in Fig. 2, and appendix Fig.
Revised: The results are given in Fig. 2, and appendix Fig.

Original: 7 and 8.
Revised: 7 and 8.

Original: We compare with two guided image translation baselines: MUNIT (Huang et al., 2018) and DRIT (Lee et al., 2018).
Revised: We compare with two guided image translation baselines: MUNIT (Huang et al., 2018) and DRIT (Lee et al., 2018).

Original: We used the published code for each method and despite our best effort, these methods fail on the task of content addition.
Revised: We used the published code for each method and despite our best effort, these methods fail on the task of content addition.

Original: In almost all cases, the baseline methods apply the style of the guide and not the added content.
Revised: In almost all cases, the baseline methods apply the style of the guide and not the added content.

Original: It should be noted that the simplicity of our approach directly translates to more efﬁcient training than the two baselines methods.
Revised: It should be noted that the simplicity of our approach directly translates to more efﬁcient training than the two baselines methods.

Original: Our method has one weighting hyperparameter, which is ﬁxed throughout the experiments.
Revised: Our method has one weighting hyperparameter, which is ﬁxed throughout the experiments.

Original: MUNIT and DRIT each has several weighting hyperparameters (since they use more loss terms) and these require attention and, need to change between the experiments, both in our runs and in the authors’ own experiments.
Revised: MUNIT and DRIT each has several weighting hyperparameters (since they use more loss terms) and these require attention and, need to change between the experiments, both in our runs and in the authors’ own experiments.

Original: In addition, our method has a lower memory footprint and a much shorter duration of each iteration.
Revised: In addition, our method has a lower memory footprint and a much shorter duration of each iteration.

Original: The statistics are reported in Tab. 2 and as can be seen, the runtime and memory footprint of our method are much closer to the Fader network by Lample et al.
Revised: The statistics are reported in Tab. 2 and as can be seen, the runtime and memory footprint of our method are much closer to the Fader network by Lample et al.

Original: (2017), which cannot perform guided mapping, than to MUNIT and DRIT.
Revised: (2017), which cannot perform guided mapping, than to MUNIT and DRIT.

Original: Since the performance of the baselines is clearly inferior in the current setting, we did not hold a user study comparing different algorithms.
Revised: Since the performance of the baselines is clearly inferior in the current setting, we did not hold a user study comparing different algorithms.

Original: Instead, we compare the output of our method directly with real images.
Revised: Instead, we compare the output of our method directly with real images.

Original: Two experiments are conducted: (i) can users tell the difference between an image from domain B and an image from domain A that was translated to domain B , and (ii) can users tell  the difference between an image from domain B and the same image, after replacing the attribute’s content (glasses, smile, or facial-hair) with that of another image from B .
Revised: Two experiments are conducted: (i) can users tell the difference between an image from domain B and an image from domain A that was translated to domain B , and (ii) can users tell [Figure] the difference between an image from domain B and the same image, after replacing the attribute’s content (glasses, smile, or facial-hair) with that of another image from B .

Original: The experiment was performed with n = 30 users, who observed 10 pairs of images each, for each of the tests.[Figure]
Revised: The experiment was performed with n = 30 users, who observed 10 pairs of images each, for each of the tests.

Original: The results are reported in Tab. 3.
Revised: The results are reported in Tab. 3.

Original: As can be seen, users are able to detect the real image over the generated one, in most of the cases.
Revised: As can be seen, users are able to detect the real image over the generated one, in most of the cases.

Original: However, the success ratio varies between the three image translation tasks and between the two types of comparisons.
Revised: However, the success ratio varies between the three image translation tasks and between the two types of comparisons.

Original: The most successful experiments, i.e., those where the users were confused the most, were in the facial hair (“beard”) category.
Revised: The most successful experiments, i.e., those where the users were confused the most, were in the facial hair (“beard”) category.

Original: In contrast, when replacing a person’s glasses with those of a random person, the users were able to tell the real image 74% of the time.
Revised: In contrast, when replacing a person’s glasses with those of a random person, the users were able to tell the real image 74% of the time.

Original: Fig.
Revised: Fig.

Original: 3 and appendix Fig.
Revised: 3 and appendix Fig.

Original: 9 and 10 show the type of images shown in the experiment where users were asked to tell an image from domain B from an hybrid image that contains a face of one image from this domain, and the attribute content from another image from it.
Revised: 9 and 10 show the type of images shown in the experiment where users were asked to tell an image from domain B from an hybrid image that contains a face of one image from this domain, and the attribute content from another image from it.

Original: As can be seen, most mixand-match combinations seem natural.
Revised: As can be seen, most mixand-match combinations seem natural.

Original: However, going over the rows, which should have a ﬁxed attribute (e.g., the same glasses), one observes some variation.
Revised: However, going over the rows, which should have a ﬁxed attribute (e.g., the same glasses), one observes some variation.

Original: This unwanted variation arises from the need to ﬁt the content to the new face.
Revised: This unwanted variation arises from the need to ﬁt the content to the new face.

Original: The method does have, as can be expected, difﬁculty dealing with low quality inputs.
Revised: The method does have, as can be expected, difﬁculty dealing with low quality inputs.

Original: Examples are shown in Fig. 4, including a misaligned source or guide image.
Revised: Examples are shown in Fig. 4, including a misaligned source or guide image.

Original: Also shown is an example in which the added content in the target domain is very subtle.
Revised: Also shown is an example in which the added content in the target domain is very subtle.

Original: These challenges result in a lower quality output.
Revised: These challenges result in a lower quality output.

Original: However, the output in each case does indicate some ability to overcome the challenging input.
Revised: However, the output in each case does indicate some ability to overcome the challenging input.

Original: To evaluate the linearity of the latent representation e 2 ( b ) , we performed interpolation experiments.
Revised: To evaluate the linearity of the latent representation e 2 ( b ) , we performed interpolation experiments.

Original: The results are presented in Fig.
Revised: The results are presented in Fig.

Original: As can be seen, the change is gradual as we interpolatelinearly between the e 2 encoding of the two guide images shown on the left and on the right.
Revised: As can be seen, the change is gradual as we interpolate

Original: In the supplementary appendix, we provide many more translation examples, see Fig.
Revised: In the supplementary appendix, we provide many more translation examples, see Fig.

Original: 11, 12, and 13.
Revised: 11, 12, and 13.

Original: While the previous experiments focused on the guided addition of content (mapping from A to B ), our method can also be applied in the other direction, from B to A .
Revised: [Table] [Table] [Table]

Original: This way, the speciﬁc content Forced choice performed by
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Selected b over g ( e 1 ( a ) , e 2 ( b (cid:48) )) , for a

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: While the previous experiments focused on the guided addition of content (mapping from A to B ), our method can also be applied in the other direction, from B to A .

Original: [Table] [Table] [Table] in image b ∈ B is removed.
Revised: This way, the speciﬁc content in image b ∈ B is removed.

Original: In our method, this is achieved simply by decoding a representation of the 
Revised: In our method, this is achieved simply by decoding a representation of the form ( e 1 ( b ) , 0) .

Original: The advantage of mapping in this direction is the availability of additional literature methods to compare with, since no guiding is necessary.
Revised: The advantage of mapping in this direction is the availability of additional literature methods to compare with, since no guiding is necessary.

Original: In Fig.
Revised: In Fig.

Original: 6, we compare the results we obtain for removing a feature with the Fader network method of Lample et al.
Revised: 6, we compare the results we obtain for removing a feature with the Fader network method of Lample et al.

Original: As can be seen, the removal process of our method results in less residuals.
Revised: As can be seen, the removal process of our method results in less residuals.

Original: To verify that we obtain a better quality in comparison with that of the published implementation of Fader networks, we have applied both an automatic classiﬁer and a user study.
Revised: To verify that we obtain a better quality in comparison with that of the published implementation of Fader networks, we have applied both an automatic classiﬁer and a user study.

Original: The classiﬁer is trained on the training set of domain A and B , using the same architecture that is used by Lample et al.
Revised: The classiﬁer is trained on the training set of domain A and B , using the same architecture that is used by Lample et al.

Original: (2017) to perform model selection.
Revised: (2017) to perform model selection.

Original: Tab. 4 presents the mean probability of class B provided by the classiﬁer for the output of both
Revised: Tab. 4 presents the mean probability of class B provided by the classiﬁer for the output of both

Original: Fader network and our method.
Revised: Fader network and our method.

Original: As can be seen, the probability to belong to the class of the image before the transformation is, as desired, low for both methods.
Revised: As can be seen, the probability to belong to the class of the image before the transformation is, as desired, low for both methods.

Original: It is somewhat lower on average in our method, despite the fact that our method does not use such a network during training.
Revised: It is somewhat lower on average in our method, despite the fact that our method does not use such a network during training.

Original: Guide
Revised: Guide

Original: S ou r ce
Revised: S ou r ce

Original: Guide
Revised: Guide

Original: S ou r ce
Revised: S ou r ce

Original: Guide
Revised: Guide

Original: S ou r ce
Revised: S ou r ce

Original: [Figure]
Revised: [Figure]

Original: The user study was conducted on n = 20 users, each examining 20 random test set triplets from each experiment.
Revised: The user study was conducted on n = 20 users, each examining 20 random test set triplets from each experiment.

Original: Each triplet showed the original image (with the feature) and the results of the two algorithms, where the feature is removed.
Revised: Each triplet showed the original image (with the feature) and the results of the two algorithms, where the feature is removed.

Original: The users preferred our method over fader 92% of the time for glasses removal, 89% of the time for facial hair removal, and 91% of the time for the removal of a smile.
Revised: The users preferred our method over fader 92% of the time for glasses removal, 89% of the time for facial hair removal, and 91% of the time for the removal of a smile.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: When converting between two domains, there is an inherent ambiguity that arises from the domainspeciﬁc information in the target domain.
Revised: When converting between two domains, there is an inherent ambiguity that arises from the domainspeciﬁc information in the target domain.

Original: In guided translation, the reference image in the target domain provides the missing information.
Revised: In guided translation, the reference image in the target domain provides the missing information.

Original: Previous work has focused on the missing information that is highly tied to the texture of the image.
Revised: Previous work has focused on the missing information that is highly tied to the texture of the image.

Original: For example, when translating between paintings and photos, DRIT adds considerable content from the reference photo.
Revised: For example, when translating between paintings and photos, DRIT adds considerable content from the reference photo.

Original: However, this is unstructured content, which is not well localized and is highly related to subsets of the image patches that exist in the target domain.
Revised: However, this is unstructured content, which is not well localized and is highly related to subsets of the image patches that exist in the target domain.

Original: In addition, the content from the reference photo that is out of the domain of paintings is not guaranteed to be fully present in the output.
Revised: In addition, the content from the reference photo that is out of the domain of paintings is not guaranteed to be fully present in the output.

Original: Our work focuses on transformations in which the domain speciﬁc content is well structured, and guarantees to replicate all of the domain speciﬁc information from the reference image.
Revised: Our work focuses on transformations in which the domain speciﬁc content is well structured, and guarantees to replicate all of the domain speciﬁc information from the reference image.

Original: This is done using a small number of networks and a surprisingly simple set of loss terms, which, due to the emergence of a disentangled representation, solves the problem convincingly.
Revised: This is done using a small number of networks and a surprisingly simple set of loss terms, which, due to the emergence of a disentangled representation, solves the problem convincingly.

Original: [Table]
Revised: [Table]

Original: Original
Revised: [Figure]  A CKNOWLEDGEMENTS This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC CoG 725974).

Original: [Table]
Revised: The theoretical analysis in this work is part of Tomer Galanti’s Ph.D thesis research conducted at Tel Aviv

Original: [Figure]
Revised: University

Original: A A DDITIONAL FIGURES
Revised: A A DDITIONAL FIGURES

Original: Our Method
Revised: Our Method

Original: MUNIT
Revised: MUNIT

Original: DRIT
Revised: DRIT

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Method

Original: MUNIT

Original: DRIT

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: B P RELIMINARIES B. N OTATIONS AND T ERMINOLOGY
Revised: B P RELIMINARIES B. N OTATIONS AND T ERMINOLOGY

Original: In this section we provide notations and terminology that are were not introduced in Sec.
Revised: In this section we provide notations and terminology that are were not introduced in Sec.

Original: but are necessary for the proofs of the claims in this section.
Revised: but are necessary for the proofs of the claims in this section.

Original: We say that three random variables (discrete or continuous)
Revised: We say that three random variables (discrete or continuous)

Original: X 1 , X 2 , X 3 form a Markov chain, indicated with X 1 → X 2 → X 3 , if P [ X 3 | X 2 , X 1 ] =
Revised: X 1 , X 2 , X 3 form a Markov chain, indicated with X 1 → X 2 → X 3 , if P [ X 3 | X 2 , X 1 ] =

Original: P [ X 3 | X 2 ] .
Revised: P [ X 3 | X 2 ] .

Original: The Data Processing Inequality (DPI) for a Markov chain X 1 → X 2 → X 3 ensures that I ( X 1 ; X 3 ) ≤ min ( I ( X 1 ; X 2 ) , I ( X 2 ; X 3 )) .
Revised: The Data Processing Inequality (DPI) for a Markov chain X 1 → X 2 → X 3 ensures that I ( X 1 ; X 3 ) ≤ min ( I ( X 1 ; X 2 ) , I ( X 2 ; X 3 )) .

Original: In particular, it holds for X 2 = f
Revised: In particular, it holds for X 2 = f

Original: ( X 1 ) and X 3 = g
Revised: ( X 1 ) and X 3 = g

Original: ( X 2 ) , where f, g are deterministic processes.
Revised: ( X 2 ) , where f, g are deterministic processes.

Original: We denote by x ∼ log N ( µ, σ 2 ) a random variable that is distributed by a log-normal distribution, i.e., log x ∼ N ( µ, σ 2 ) .
Revised: We denote by x ∼ log N ( µ, σ 2 ) a random variable that is distributed by a log-normal distribution, i.e., log x ∼ N ( µ, σ 2 ) .

Original: We consider that the mean and variance of a log-normal distribution log N ( µ, σ 2 ) are exp( µ + σ 2 / 2) and (exp( σ 2 ) − 1) exp(2 µ + σ 2 ) respectively.
Revised: We consider that the mean and variance of a log-normal distribution log N ( µ, σ 2 ) are exp( µ + σ 2 / 2) and (exp( σ 2 ) − 1) exp(2 µ + σ 2 ) respectively.

Original: We denote by W (cid:74) U :=
Revised: We denote by W (cid:74) U :=

Original: ( W k,j · U k,j ) k ≤ m,j ≤ m the Hadamard product of two matrices W , U ∈ R m × n .
Revised: ( W k,j · U k,j ) k ≤ m,j ≤ m the Hadamard product of two matrices W , U ∈ R m × n .

Original: For a given vector x ∈ R m , we denote dim ( x ) :
Revised: For a given vector x ∈ R m , we denote dim ( x ) :

Original: = m and for a matrix W ∈ R m × n , we denote dim ( W ) :
Revised: = m and for a matrix W ∈ R m × n , we denote dim ( W ) :

Original: = mn .
Revised: = mn .

Original: In addition, we denote x 2 = x (cid:74) x
Revised: In addition, we denote x 2 = x (cid:74) x

Original: = ( x 21 , . .
Revised: = ( x 21 , . .

Original: , x 2 m ) and E
Revised: , x 2 m ) and E

Original: [ x ] = ( E [ x 1 ] , . .
Revised: [ x ] = ( E [ x 1 ] , . .

Original: , E [ x m ]) .
Revised: , E [ x m ]) .

Original: The indicator function, is denoted by 1 1 [ x ] for a boolean variable x ∈ { true , false } (i.e., 1 1 [ x ] = 1 if x = true and 1 1 [ x ] = 0 o.w).
Revised: The indicator function, is denoted by 1 1 [ x ] for a boolean variable x ∈ { true , false } (i.e., 1 1 [ x ] = 1 if x = true and 1 1 [ x ] = 0 o.w).

Original: B. L EMMAS
Revised: B. L EMMAS

Original: In this section, we provide useful lemmas that aid in the proofs of our main results.
Revised: In this section, we provide useful lemmas that aid in the proofs of our main results.

Original: Lemma 4.
Revised: Lemma 4.

Original: Let x = ( x 1 , . . .
Revised: Let x = ( x 1 , . . .

Original: , x n ) ∈ R n be a random vector.
Revised: , x n ) ∈ R n be a random vector.

Original: Let µ 1 , . . .
Revised: Let µ 1 , . . .

Original: , µ n : R → R be continuous invertible functions and we denote µ ( x ) :
Revised: , µ n : R → R be continuous invertible functions and we denote µ ( x ) :

Original: = ( µ 1 ( x 1 ) , . .
Revised: = ( µ 1 ( x 1 ) , . .

Original: . , µ n ( x n )) .
Revised: . , µ n ( x n )) .

Original: Then, TC ( x ) =
Revised: Then, TC ( x ) =

Original: TC ( µ ( x )) .
Revised: TC ( µ ( x )) .

Original: Proof.
Revised: Proof.

Original: First, we consider that:
Revised: First, we consider that:

Original: [Equation]
Revised: [Equation]

Original: where, ¯ x := (¯ x 1 , . .
Revised: where, ¯ x := (¯ x 1 , . .

Original: , ¯ x n ) is a vector of independent random variables, such that ¯ x i is distributed, according to the marginal distribution of x i .
Revised: , ¯ x n ) is a vector of independent random variables, such that ¯ x i is distributed, according to the marginal distribution of x i .

Original: KL-divergence is invariant to applying continuous invertible transformations, i.e., D KL ( X (cid:107) Y ) =
Revised: KL-divergence is invariant to applying continuous invertible transformations, i.e., D KL ( X (cid:107) Y ) =

Original: D KL ( µ ( X ) (cid:107) µ ( Y )) for µ that is continuous and invertible.
Revised: D KL ( µ ( X ) (cid:107) µ ( Y )) for µ that is continuous and invertible.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Lemma 5.
Revised: Lemma 5.

Original: Let p ∈
Revised: Let p ∈

Original: Then, H ( p ) ≤ 2 log(2) (cid:112) p (1 − p ) .
Revised: Then, H ( p ) ≤ 2 log(2) (cid:112) p (1 − p ) .

Original: Proof.
Revised: Proof.

Original: See (https://math.stackexchange.com/users/44121/jack daurizio).
Revised: See (https://math.stackexchange.com/users/44121/jack daurizio).

Original: The following lemma is a modiﬁcation of Claim 2.1 in (Regev, 2013).
Revised: The following lemma is a modiﬁcation of Claim 2.1 in (Regev, 2013).

Original: Lemma 6.
Revised: Lemma 6.

Original: Let X
Revised: Let X

Original: and Y be two random variables.
Revised: and Y be two random variables.

Original: Assume that there is a function (i.e., a deterministic process) F , such that P [ F ( Y )
Revised: Assume that there is a function (i.e., a deterministic process) F , such that P [ F ( Y )

Original: =
Revised: =

Original: X ] ≥ q ≥ 1 / 2 .
Revised: X ] ≥ q ≥ 1 / 2 .

Original: Then, I ( X ; Y ) ≥ qH ( X ) − H ( q ) .
Revised: Then, I ( X ; Y ) ≥ qH ( X ) − H ( q ) .

Original: Proof.
Revised: Proof.

Original: By the data processing inequality,
Revised: By the data processing inequality,

Original: [Equation]
Revised: [Equation]

Original: Since conditioning does not increase entropy, H ( 1 1 X = F ( Y )
Revised: Since conditioning does not increase entropy, H ( 1 1 X = F ( Y )

Original: | F ( Y )) ≤ 
Revised: | F ( Y )) ≤ H

Original: [Equation]
Revised: [Equation]

Original: H
Revised: X

Original: [Equation]
Revised: [Equation]

Original:  q ) .
Revised: F ( Y ) ) ≤ H ( q ) .

Original: In addition,
Revised: In addition,

Original: [Equation]
Revised: [Equation]

Original: Therefore, we conclude that, I ( X ; Y ) ≥ qH ( X ) − H ( q ) .
Revised: Therefore, we conclude that, I ( X ; Y ) ≥ qH ( X ) − H ( q ) .

Original: Lemma 7.
Revised: Lemma 7.

Original: Let b ∼ D B be a distribution over a discrete set X B ⊂ R M and h v : R M → R M is a (possibly random) function.
Revised: Let b ∼ D B be a distribution over a discrete set X B ⊂ R M and h v : R M → R M is a (possibly random) function.

Original: Assume that ∀ x 1 (cid:54) = x 2 ∈ X B : (cid:107) x 1 − x 2 (cid:107) 1 > ∆ .
Revised: Assume that ∀ x 1 (cid:54) = x 2 ∈ X B : (cid:107) x 1 − x 2 (cid:107) 1 > ∆ .

Original: Let P [ (cid:107) h v ( b )
Revised: Let P [ (cid:107) h v ( b )

Original: − b (cid:107) 1 ≤ ∆] ≥ q ≥ 1 / 2 .
Revised: − b (cid:107) 1 ≤ ∆] ≥ q ≥ 1 / 2 .

Original: Then, I ( h v ( b ); b ) ≥ qH ( b ) − H ( q ) .
Revised: Then, I ( h v ( b ); b ) ≥ qH ( b ) − H ( q ) .

Original: Proof.
Revised: Proof.

Original: Let F ( u ) :=
Revised: Let F ( u ) :=

Original: arg min x ∈ X B (cid:107) u − x (cid:107) 1 .
Revised: arg min x ∈ X B (cid:107) u − x (cid:107) 1 .

Original: Since the members of X B are ∆ -distant from each other, if (cid:107) h v ( b ) − b (cid:107) 1 ≤ ∆ , then, F ( h v ( b ))
Revised: Since the members of X B are ∆ -distant from each other, if (cid:107) h v ( b ) − b (cid:107) 1 ≤ ∆ , then, F ( h v ( b ))

Original: =
Revised: =

Original: b .
Revised: b .

Original: Therefore, we have:
Revised: Therefore, we have:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By Lem.
Revised: By Lem.

Original: 6, for X : ← b , Y : ← h v ( b ) and F : ← F , we have: I ( h v ( b ); b ) ≥ qH ( b ) − H ( q ) .
Revised: 6, for X : ← b , Y : ← h v ( b ) and F : ← F , we have: I ( h v ( b ); b ) ≥ qH ( b ) − H ( q ) .

Original: The following lemma is an example of three uncorrelated variables X, Y, Z , such that there is a dimensionality reducing linear transformation over them that preserves all of their information.
Revised: The following lemma is an example of three uncorrelated variables X, Y, Z , such that there is a dimensionality reducing linear transformation over them that preserves all of their information.

Original: Lemma 8.
Revised: Lemma 8.

Original: Let X
Revised: Let X

Original: and Y be two independent uniform distributions over [ − 1 , 1] and Z =
Revised: and Y be two independent uniform distributions over [ − 1 , 1] and Z =

Original: ( X + Y ) 2 .
Revised: ( X + Y ) 2 .

Original: Then, the transformation T ( x, y, z ) =
Revised: Then, the transformation T ( x, y, z ) =

Original: ( x, y ) satisﬁes
Revised: ( x, y ) satisﬁes

Original: I ( X, Y, Z ; T ( X, Y, Z ))
Revised: I ( X, Y, Z ; T ( X, Y, Z ))

Original: =
Revised: =

Original: H ( X, Y, Z ) and
Revised: H ( X, Y, Z ) and

Original: Cov ( X, Y ) =
Revised: Cov ( X, Y ) =

Original: Cov ( X, Z ) =
Revised: Cov ( X, Z ) =

Original: Cov ( Y, Z ) .
Revised: Cov ( Y, Z ) .

Original: Proof.
Revised: Proof.

Original: Since X and Y are independent, their covariance is zero.
Revised: Since X and Y are independent, their covariance is zero.

Original: By the deﬁnition of X and Y , we have: E [ X ] =
Revised: By the deﬁnition of X and Y , we have: E [ X ] =

Original: E
Revised: E

Original: [ Y ] =
Revised: [ Y ] =

Original: E [ X 3 ] = 0 .
Revised: E [ X 3 ] = 0 .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Finally, we consider that T is a homeomorphic transformation T : ( x, y, ( x + y ) 2 ) (cid:55)→ ( x, y ) (between the manifolds { ( x, y, z ) | x, y ∈
Revised: Finally, we consider that T is a homeomorphic transformation T : ( x, y, ( x + y ) 2 ) (cid:55)→ ( x, y ) (between the manifolds { ( x, y, z ) | x, y ∈

Original: [ − 1 , 1] , z = ( x + y ) 2 } and [ − 1 , 1] 2 ) and mutual information is invariant to applications of homeomorphic transformations, i.e., I ( X ; Y ) =
Revised: [ − 1 , 1] , z = ( x + y ) 2 } and [ − 1 , 1] 2 ) and mutual information is invariant to applications of homeomorphic transformations, i.e., I ( X ; Y ) =

Original: I ( µ ( X ); ν ( Y )) for homeomorphisms µ and ν over the sample spaces of X and Y (resp.).
Revised: I ( µ ( X ); ν ( Y )) for homeomorphisms µ and ν over the sample spaces of X and Y (resp.).

Original: Therefore,
Revised: Therefore,

Original: I ( X, Y, Z ; T ( X, Y, Z ))
Revised: I ( X, Y, Z ; T ( X, Y, Z ))

Original: =
Revised: =

Original: I ( X, Y, Z ; X, Y, Z ) =
Revised: I ( X, Y, Z ; X, Y, Z ) =

Original: H ( X, Y, Z ) .
Revised: H ( X, Y, Z ) .

Original: C P ROOFS OF THE M AIN R ESULTS
Revised: C P ROOFS OF THE M AIN R ESULTS

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume that the loss function (cid:96) is symmetric and obeys the triangle inequality.
Revised: Assume that the loss function (cid:96) is symmetric and obeys the triangle inequality.

Original: Then, for any autoencoder h = g ◦ f ∈ H , such that f ( x 1 , x 2 )
Revised: Then, for any autoencoder h = g ◦ f ∈ H , such that f ( x 1 , x 2 )

Original: = ( e 1 ( x 1 ) , e 2 ( x 2 ))
Revised: = ( e 1 ( x 1 ) , e 2 ( x 2 ))

Original: ∈ F is an encoder and g ∈ M is a decoder, the following holds,
Revised: ∈ F is an encoder and g ∈ M is a decoder, the following holds,

Original: [Equation]
Revised: [Equation]

Original: where D B, ˆ B is the distribution of ( b , b ) where b ∼ D B .
Revised: where D B, ˆ B is the distribution of ( b , b ) where b ∼ D B .

Original: Proof.
Revised: Proof.

Original: Let g ∗ ∈ arg min g ∈M (cid:110) R D A,B [ g ◦ f, y ] + R D B, ˆ B [ g ◦ f, y ] (cid:111) .
Revised: Let g ∗ ∈ arg min g ∈M (cid:110) R D A,B [ g ◦ f, y ] + R D B, ˆ B [ g ◦ f, y ] (cid:111) .

Original: Since the loss (cid:96) obeys the triangle inequality,
Revised: Since the loss (cid:96) obeys the triangle inequality,

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of discrepancy,
Revised: By the deﬁnition of discrepancy,

Original: [Equation]
Revised: [Equation]

Original: Again, by the triangle inequality,
Revised: Again, by the triangle inequality,

Original: [Equation]
Revised: [Equation]

Original: Lemma 1.
Revised: Lemma 1.

Original: Let M be the set of neural networks of the form: c ( x ) =
Revised: Let M be the set of neural networks of the form: c ( x ) =

Original: φ ( W r . . .
Revised: φ ( W r . . .

Original: φ ( W 2 φ ( W 1 x + q ))) , where, W i ∈ R d i × d i +1 for i ∈ { 1 , . . .
Revised: φ ( W 2 φ ( W 1 x + q ))) , where, W i ∈ R d i × d i +1 for i ∈ { 1 , . . .

Original: , r − 1 } , q ∈ R d 2 and d 1 = E 1 + E 2 .
Revised: , r − 1 } , q ∈ R d 2 and d 1 = E 1 + E 2 .

Original: In addition, φ ( x 1 , . . .
Revised: In addition, φ ( x 1 , . . .

Original: , x k ) =
Revised: , x k ) =

Original: ( φ 1 ( x 1 ) , . . .
Revised: ( φ 1 ( x 1 ) , . . .

Original: , φ 1 ( x k )) , for k ∈ N , ( x 1 , . . .
Revised: , φ 1 ( x k )) , for k ∈ N , ( x 1 , . . .

Original: , x k ) ∈ R k and a non-linear activation function φ 1 : R → R .
Revised: , x k ) ∈ R k and a non-linear activation function φ 1 : R → R .

Original: Let M (cid:48) be the same as M with d 1 = E 1 (instead of d 1 = E 1 + E 2 ).
Revised: Let M (cid:48) be the same as M with d 1 = E 1 (instead of d 1 = E 1 + E 2 ).

Original: Let f ( x )
Revised: Let f ( x )

Original: =
Revised: =

Original: ( e 1 ( x ) , e 2 ( x )) be an encoder and assume that: e 1 ( b ) | =
Revised: ( e 1 ( x ) , e 2 ( x )) be an encoder and assume that: e 1 ( b ) | =

Original: e 2 ( b ) .
Revised: e 2 ( b ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the deﬁnition of discrepancy, and since e 1 ( b ) | = e 2 ( b ) , we have:
Revised: By the deﬁnition of discrepancy, and since e 1 ( b ) | = e 2 ( b ) , we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: By | E [ x ] | ≤ E [ | x | ] and E x [sup y ∈ Y f ( x , y )]
Revised: By | E [ x ] | ≤ E [ | x | ] and E x [sup y ∈ Y f ( x , y )]

Original: ≤ sup y ∈
Revised: ≤ sup y ∈

Original: Y E
Revised: Y E

Original: x [ f ( x , y )] , we have:
Revised: x [ f ( x , y )] , we have:

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of M
Revised: By the deﬁnition of M

Original: [Equation]
Revised: [Equation]

Original: for any c ∈ M and a ﬁxed vector
Revised: for any c ∈ M and a ﬁxed vector

Original: [Equation]
Revised: [Equation]

Revised: y ∈ R E

Original: y ∈ R E 2 , there is a function u ∈ M (cid:48) , such that for every x ∈ R E 1 , we have: c ( x , y )
Revised: [Equation] there is a function u ∈ M (cid:48) , such that for every x ∈ R E [Equation] we have: c ( x , y )

Original: =
Revised: =

Original: u ( x ) .
Revised: u ( x ) .

Original: Therefore, we can rewrite the last equation as follows:
Revised: Therefore, we can rewrite the last equation as follows:

Original: [Equation]
Revised: [Equation]

Original: C. E MERGENCE OF D ISENTANGLED R EPRESENTATIONS
Revised: C. E MERGENCE OF D ISENTANGLED R EPRESENTATIONS

Original: In this section, we employ the theory of (Achille & Soatto, 2018) in order to show the emergence of disentangled representations, when an autoencoder h
Revised: In this section, we employ the theory of (Achille & Soatto, 2018) in order to show the emergence of disentangled representations, when an autoencoder h

Original: =
Revised: =

Original: g ◦ f generalizes well.
Revised: g ◦ f generalizes well.

Original: Our analysis shows that by learning an autoencoder such that the encoder f has log-normal regularization, there is a high likelihood of learning disentangled representations.
Revised: Our analysis shows that by learning an autoencoder such that the encoder f has log-normal regularization, there is a high likelihood of learning disentangled representations.

Original: We note that until now, we treated the autoencoder h as a function of two variables ( a , b or b , b ).
Revised: We note that until now, we treated the autoencoder h as a function of two variables ( a , b or b , b ).

Original: From now on, if the two variables are the same, then, we simply write h v ( b ) .
Revised: From now on, if the two variables are the same, then, we simply write h v ( b ) .

Original: In addition, by Eq. 11, instead of writing, R D B, ˆ B [ g ◦ f, y ] we can simply write R D B [ h, Id ] :=
Revised: In addition, by Eq. 11, instead of writing, R D B, ˆ B [ g ◦ f, y ] we can simply write R D B [ h, Id ] :=

Original: E f E b ∼ D B [ (cid:96) ( g ◦ f ( b ) , b )] .
Revised: E f E b ∼ D B [ (cid:96) ( g ◦ f ( b ) , b )] .

Original: The framework of Achille & Soatto (2018) relies on a few assumptions, which are imported to our case.
Revised: The framework of Achille & Soatto (2018) relies on a few assumptions, which are imported to our case.

Original: The algorithm is provided with m i.i.d samples S B = { b i } mi =1 from the distribution
Revised: The algorithm is provided with m i.i.d samples S B = { b i } mi =1 from the distribution

Original: D B and trains an encoder-decoder neural network h v =
Revised: D B and trains an encoder-decoder neural network h v =

Original: g u ◦ f w , such that the parameters of the encoder, f v , have log-normal perturbations.
Revised: g u ◦ f w , such that the parameters of the encoder, f v , have log-normal perturbations.

Original: Formally, we learn a mapping h v of the form h v ( x ) =
Revised: Formally, we learn a mapping h v of the form h v ( x ) =

Original: φ ( W 2 t φ ( W 2 t − 1 . . .
Revised: φ ( W 2 t φ ( W 2 t − 1 . . .

Original: φ ( W 1 x ))) , where
Revised: φ ( W 1 x ))) , where

Original: W i ∈ R
Revised: W i ∈ R

Original: d i +1 ×
Revised: d i +1 ×

Original: d i , for i ∈ { 1 , . . .
Revised: d i , for i ∈ { 1 , . . .

Original: , 2 t − 1 } , d 1 = d 2 t = M , w =
Revised: , 2 t − 1 } , d 1 = d 2 t = M , w =

Original: ( W t , . . .
Revised: ( W t , . . .

Original: , W 1 ) , u = ( W 2 t , . . .
Revised: , W 1 ) , u = ( W 2 t , . . .

Original: , W t +1 ) and v = ( W 2 t , . . .
Revised: , W t +1 ) and v = ( W 2 t , . . .

Original: , W 1 ) .
Revised: , W 1 ) .

Original: Here, φ ( x 1 , . . . ,
Revised: Here, φ ( x 1 , . . . ,

Original: x m ) =
Revised: x m ) =

Original: ( φ 1 ( x 1 ) , . . .
Revised: ( φ 1 ( x 1 ) , . . .

Original: , φ 1 ( x k )) is a non-linear activation function φ 1 : R → R extended for all m ∈ N and ( x 1 , . . .
Revised: , φ 1 ( x k )) is a non-linear activation function φ 1 : R → R extended for all m ∈ N and ( x 1 , . . .

Original: , x k ) ∈ R k .
Revised: , x k ) ∈ R k .

Original: We assume that φ 1 : R → R is a homeomorphism (i.e., φ 1 is invertible, continuous and φ − 1is also continuous).
Revised: We assume that φ 1 : R → R is a homeomorphism (i.e., φ 1 is invertible, continuous and φ − 1is also continuous).

Original: The encoder f w is the composition of the ﬁrst t layers and the decoder g u is composed of the last t layers of h v .
Revised: The encoder f w is the composition of the ﬁrst t layers and the decoder g u is composed of the last t layers of h v .

Original: Following Achille & Soatto (2018), we assume that the posterior distribution p ( W ki,j | S B ) is deﬁned as a Gaussian dropout,follows:
Revised: Following Achille & Soatto (2018), we assume that the posterior distribution p ( W ki,j | S B ) is deﬁned as a Gaussian dropout,

Original: [Equation]
Revised: [Equation]

Original:  where (cid:15) i ∈ R
Revised: 27) where (cid:15) i ∈ R

Original: d i +1 ×
Revised: d i +1 ×

Original: d i and (cid:15) ki,j ∼ log N ( − α/ 2 , α ) , for k ∈ { 1 , . . .
Revised: d i and (cid:15) ki,j ∼ log N ( − α/ 2 , α ) , for k ∈ { 1 , . . .

Original: , t } .
Revised: , t } .

Original: We consider that the mean and variance of log N ( − α/ 2 , α ) are 1 and exp( α ) − 1 respectively.
Revised: We consider that the mean and variance of log N ( − α/ 2 , α ) are 1 and exp( α ) − 1 respectively.

Original: Here, ˆ W k is a learned mean of the k ’th layer of the encoder.
Revised: Here, ˆ W k is a learned mean of the k ’th layer of the encoder.

Original: We denote the output of the k ’th layer of f w by z k , i.e., z k = φ ( W k φ ( W k − 1 . . .
Revised: We denote the output of the k ’th layer of f w by z k , i.e., z k = φ ( W k φ ( W k − 1 . . .

Original: φ ( W 1 x ))) .
Revised: φ ( W 1 x ))) .

Original: Implicit Emergence of Disentangled Representations
Revised: Implicit Emergence of Disentangled Representations

Original: The following lemma is a corollary of
Revised: The following lemma is a corollary of

Original: Proposition 5.2 in (Achille & Soatto, 2018) for our model.
Revised: Proposition 5.2 in (Achille & Soatto, 2018) for our model.

Original: Lemma 9.
Revised: Lemma 9.

Original: Let k ∈ { 1 , . .
Revised: Let k ∈ { 1 , . .

Original: . , t } , y k
Revised: . , t } , y k

Original: = W k φ ( W k − 1 . . .
Revised: = W k φ ( W k − 1 . . .

Original: φ ( W 1 b )) and z k = φ ( y k ) , for W k = (cid:15)
Revised: φ ( W 1 b )) and z k = φ ( y k ) , for W k = (cid:15)

Original: k
Revised: k

Original: (cid:74)
Revised: (cid:74)

Original: ˆ W k , where (cid:15) ki,j ∼ log N ( α k / 2 , α k ) .
Revised: ˆ W k , where (cid:15) ki,j ∼ log N ( α k / 2 , α k ) .

Original: Further, assume that the marginals of P [ y k ] and
Revised: Further, assume that the marginals of P [ y k ] and

Original: P [ y k | z k − 1 ] are Gaussians, the components of z k − 1 are uncorrelated and that their kurtosis is uniformly bounded.
Revised: P [ y k | z k − 1 ] are Gaussians, the components of z k − 1 are uncorrelated and that their kurtosis is uniformly bounded.

Original: Let q ( α ) :
Revised: Let q ( α ) :

Original: = − 12 log (1 − exp ( − α )) .
Revised: = − 12 log ( [Equation] exp ( − α )) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Proposition 5.2 in Achille & Soatto (2018), we have:
Revised: By Proposition 5.2 in Achille & Soatto (2018), we have:

Original: [Equation]
Revised: [Equation]

Original: By Lem.
Revised: By Lem.

Original: 4, we have, TC
Revised: 4, we have, TC

Original: ( y k ) =
Revised: ( y k ) =

Original: TC ( z k ) .
Revised: TC ( z k ) .

Original: In addition, dim ( y k ) = dim ( z k ) .
Revised: In addition, dim ( y k ) = dim ( z k ) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Finally, by the data processing inequality, for X 1 := b , X 2 :=
Revised: Finally, by the data processing inequality, for X 1 := b , X 2 :=

Original: z k − 1 , X 3 := y k and X 4 :
Revised: z k − 1 , X 3 := y k and X 4 :

Original: = h v ( b ) , we have,
Revised: = h v ( b ) , we have,

Original: I ( h v ( b ); b ) ≤ I ( y k ; z k − 1 ) .
Revised: I ( h v ( b ); b ) ≤ I ( y k ; z k − 1 ) .

Original: The bound follows from Eq.
Revised: The bound follows from Eq.

Original: 30 and the last observation.
Revised: 30 and the last observation.

Original: Lem.
Revised: Lem.

Original: 9 provides an upper bound on the total correlation of the k ’th layer of the autoencoder h v ( b ) .
Revised: 9 provides an upper bound on the total correlation of the k ’th layer of the autoencoder h v ( b ) .

Original: This bound assumes that the marginal distributions of y k and y k | z k − 1 are Gaussians.
Revised: This bound assumes that the marginal distributions of y k and y k | z k − 1 are Gaussians.

Original: This is a reasonable assumption if dim ( z k − 1 ) is large by the central limit theorem.
Revised: This is a reasonable assumption if dim ( z k − 1 ) is large by the central limit theorem.

Original: We also assume that there are no pair-wise linear correlations between the components of z k − 1 .
Revised: We also assume that there are no pair-wise linear correlations between the components of z k − 1 .

Original: In some sense, this assumption can be viewed as minimality of z k − 1 .
Revised: In some sense, this assumption can be viewed as minimality of z k − 1 .

Original: Informally, if there is a strong linear correlation between two components of z k − 1 , then, we can throw away one of them and keep most of the information.
Revised: Informally, if there is a strong linear correlation between two components of z k − 1 , then, we can throw away one of them and keep most of the information.

Original: On the other hand, if the components of z k − 1 are uncorrelated, the existence of a dimensionality reducing linear transformation W
Revised: On the other hand, if the components of z k − 1 are uncorrelated, the existence of a dimensionality reducing linear transformation W

Original: such that
Revised: such that

Original: I ( W z k − 1 ; z k − 1 ) =
Revised: I ( W z k − 1 ; z k − 1 ) =

Original: H ( z k − 1 ) is still a possibility (for a concrete example, see Lem.
Revised: H ( z k − 1 ) is still a possibility (for a concrete example, see Lem.

Original: Hence, the next layer, z k , is still useful, in order to compress the representation and can still preserve all the information.
Revised: Hence, the next layer, z k , is still useful, in order to compress the representation and can still preserve all the information.

Original: We also assume that the kurtosis of the components of z k − 1 is uniformly bounded.
Revised: We also assume that the kurtosis of the components of z k − 1 is uniformly bounded.

Original: This is a technical hypothesis that is always satisﬁed, if the components of z k − 1 are sub-Gaussian or with uniformly bounded support.
Revised: This is a technical hypothesis that is always satisﬁed, if the components of z k − 1 are sub-Gaussian or with uniformly bounded support.

Original: The function q ( α k ) is monotonically increasing as α k tends to 0 .
Revised: The function q ( α k ) is monotonically increasing as α k tends to 0 .

Original: Therefore, this bound realizes a trade-off between the mutual information of h v ( b ) and b and the amount of perturbations in the encoder.
Revised: Therefore, this bound realizes a trade-off between the mutual information of h v ( b ) and b and the amount of perturbations in the encoder.

Original: If the perturbations in the encoder are stronger, then, the ability of the autoencoder to reconstruct b decays.
Revised: If the perturbations in the encoder are stronger, then, the ability of the autoencoder to reconstruct b decays.

Original: On the other hand, if the perturbations are small, then, q ( α ) increases.
Revised: On the other hand, if the perturbations are small, then, q ( α ) increases.

Original: We consider that the bound decreases, as I ( h v ( b ); b ) increases.
Revised: We consider that the bound decreases, as I ( h v ( b ); b ) increases.

Original: It is reasonable to believe that since h v ( b ) is an autoencoder that is being trained to reconstruct b , I ( h v ( b ); b ) is maximized implicitly.
Revised: It is reasonable to believe that since h v ( b ) is an autoencoder that is being trained to reconstruct b , I ( h v ( b ); b ) is maximized implicitly.

Original: The problem of training an autoencoder that maintains a reconstruction that has high mutual information with respect to the input has recently attracted a considerable attention.
Revised: The problem of training an autoencoder that maintains a reconstruction that has high mutual information with respect to the input has recently attracted a considerable attention.

Original: Several methods for maximizing this term explicitly were proposed in (Zhao et al., 2017; Phuong et al., 2018).
Revised: Several methods for maximizing this term explicitly were proposed in (Zhao et al., 2017; Phuong et al., 2018).

Original: In the following lemma, we show that if the samples of D B are well separated, then, the mutual information
Revised: In the following lemma, we show that if the samples of D B are well separated, then, the mutual information

Original: I ( h v ( b ); b ) is implicitly maximized as the reconstruction, h v ( b )
Revised: I ( h v ( b ); b ) is implicitly maximized as the reconstruction, h v ( b )

Original: ≈ b improves.
Revised: ≈ b improves.

Original: Lemma 10.
Revised: Lemma 10.

Original: Let b ∼ D B be a random vector over a discrete set X B ⊂ R M and h v : R M → R M an autoencoder.
Revised: Let b ∼ D B be a random vector over a discrete set X B ⊂ R M and h v : R M → R M an autoencoder.

Original: Assume that ∀ x 1 (cid:54) = x 2 ∈ X B : (cid:107) x 1 − x 2 (cid:107) 1 > ∆ and R D B [ h v ( b ) , Id ] ≤ ∆2 .
Revised: Assume that ∀ x 1 (cid:54) = x 2 ∈ X B : (cid:107) x 1 − x 2 (cid:107) 1 > ∆ and R D B [ h v ( b ) , Id ] ≤ ∆2 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, assume by contradiction that:
Revised: First, assume by contradiction that:

Original: [Equation]
Revised: [Equation]

Original: In other words [Equation] P [ (cid:107) h v ( b ) − b (cid:107) 1 > ∆] > R D B [ h v ( b ) , Id ] / ∆ .
Revised: In other words, P [ (cid:107) h v ( b ) − b (cid:107) 1 > ∆] > R D B [ h v ( b ) , Id ] / ∆ .

Original: In particular, we arrive at a contradiction:
Revised: In particular, we arrive at a contradiction:

Original: [Equation]
Revised: [Equation]

Original: By the above contradiction and the hypothesis that R DB [ h v ( b ) , Id ] ∆ ≤ 1 / 2 , we have, q ≥ 
Revised: By the above contradiction and the hypothesis that R DB [ h v ( b ) , Id ] ∆ ≤ 1 / 2 , we have, q ≥ [Equation] [Equation] Id

Original: [Equation]
Revised: [Equation]

Original: Therefore, by Lem
Revised: Therefore, by Lem

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Additionally, by Lem. 5, we have:
Revised: Additionally, by Lem. 5, we have:

Original: [Equation]
Revised: [Equation]

Original: Finally,
Revised: Finally,

Original: [Equation]
Revised: [Equation]

Original: The above lemma asserts that if the samples in D B are well separated, whenever the autoencoder has a small expected reconstruction error R D B [ h v ( b ) , Id ] , then, the mutual information I ( h v ( b ); b ) is at least a large portion of H ( b ) .
Revised: The above lemma asserts that if the samples in D B are well separated, whenever the autoencoder has a small expected reconstruction error R D B [ h v ( b ) , Id ] , then, the mutual information I ( h v ( b ); b ) is at least a large portion of H ( b ) .

Original: Therefore, we conclude that if the autoencoder generalizes well, then, it also maximizes the mutual information I ( h v ( b ); b ) .
Revised: Therefore, we conclude that if the autoencoder generalizes well, then, it also maximizes the mutual information I ( h v ( b ); b ) .

Original: Finally, we note that we cannot directly apply Lem.
Revised: Finally, we note that we cannot directly apply Lem.

Original: 10 to bound the mutual information in Lem.
Revised: 10 to bound the mutual information in Lem.

Original: That is because, in Lem.
Revised: That is because, in Lem.

Original: 9, we assume that the components of y k are distributed normally, which implies that the distribution of b must be continuous.
Revised: 9, we assume that the components of y k are distributed normally, which implies that the distribution of b must be continuous.

Original: On the other hand, in Lem. 10 we assume that b is distributed according to a discrete distribution.
Revised: On the other hand, in Lem. 10 we assume that b is distributed according to a discrete distribution.

Original: In order to reduce this friction, instead of assuming that the each component of z k is distributed according to a normal distribution, we can assume that it is distributed according to a discrete approximation of a normal distribution.
Revised: In order to reduce this friction, instead of assuming that the each component of z k is distributed according to a normal distribution, we can assume that it is distributed according to a discrete approximation of a normal distribution.