Original: G ENERATING F URRY C ARS : D ISENTANGLING O BJECT S HAPE & A PPEARANCE ACROSS M ULTIPLE D OMAINS
Revised: G ENERATING F URRY C ARS : D ISENTANGLING O BJECT S HAPE & A PPEARANCE ACROSS M ULTIPLE D OMAINS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider the novel task of learning disentangled representations of object shape and appearance across multiple domains (e.g., dogs and cars).
Revised: We consider the novel task of learning disentangled representations of object shape and appearance across multiple domains (e.g., dogs and cars).

Original: The goal is to learn a generative model that learns an intermediate distribution, which borrows a subset of properties from each domain, enabling the generation of images that did not exist in any domain exclusively.
Revised: The goal is to learn a generative model that learns an intermediate distribution, which borrows a subset of properties from each domain, enabling the generation of images that did not exist in any domain exclusively.

Original: This challenging problem requires an accurate disentanglement of object shape, appearance, and background from each domain, so that the appearance and shape factors from the two domains can be interchanged.
Revised: This challenging problem requires an accurate disentanglement of object shape, appearance, and background from each domain, so that the appearance and shape factors from the two domains can be interchanged.

Original: We augment an existing approach that can disentangle factors within a single domain but struggles to do so across domains.
Revised: We augment an existing approach that can disentangle factors within a single domain but struggles to do so across domains.

Original: Our key technical contribution is to represent object appearance with a differentiable histogram of visual features, and to optimize the generator so that two images with the same latent appearance factor but different latent shape factors produce similar histograms.
Revised: Our key technical contribution is to represent object appearance with a differentiable histogram of visual features, and to optimize the generator so that two images with the same latent appearance factor but different latent shape factors produce similar histograms.

Original: On multiple multi-domain datasets, we demonstrate our method leads to accurate and consistent appearance and shape transfer across domains.
Revised: On multiple multi-domain datasets, we demonstrate our method leads to accurate and consistent appearance and shape transfer across domains.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Humans possess the incredible ability of being able to combine properties from multiple image distributions to create entirely new visual concepts.
Revised: Humans possess the incredible ability of being able to combine properties from multiple image distributions to create entirely new visual concepts.

Original: For example, Lake et al.
Revised: For example, Lake et al.

Original: (2015) discussed how humans can parse different object parts (e.g., wheels of a car, handle of a lawn mower) and combine them to conceptualize novel object categories (a scooter).
Revised: (2015) discussed how humans can parse different object parts (e.g., wheels of a car, handle of a lawn mower) and combine them to conceptualize novel object categories (a scooter).

Original: Fig.
Revised: Fig.

Original: 2 illustrates another example from a different angle; it is easy for us humans to imagine how the brown car would look if its appearance were borrowed from the blue and red bird.
Revised: 2 illustrates another example from a different angle; it is easy for us humans to imagine how the brown car would look if its appearance were borrowed from the blue and red bird.

Original: To model a similar ability in machines, a precise disentanglement of shape and appearance features, and the ability to combine them across different domains are needed.
Revised: To model a similar ability in machines, a precise disentanglement of shape and appearance features, and the ability to combine them across different domains are needed.

Original: In this work, we seek to develop a framework to do just that, where we deﬁne domains to correspond to “basic-level categories” (Rosch, 1978).
Revised: In this work, we seek to develop a framework to do just that, where we deﬁne domains to correspond to “basic-level categories” (Rosch, 1978).

Original: Disentangling the factors of variation in visual data has received signiﬁcant attention (Chen et al., 2016;
Revised: Disentangling the factors of variation in visual data has received signiﬁcant attention (Chen et al., 2016;

Original: Higgins et al., 2017; Denton & Birodkar, 2017; Singh et al., 2019), in particular with advances in generative models (Goodfellow et al., 2014; Radford et al., 2016;
Revised: Higgins et al., 2017; Denton & Birodkar, 2017; Singh et al., 2019), in particular with advances in generative models (Goodfellow et al., 2014; Radford et al., 2016;

Original: Zhang et al., 2018; Karras et al.,
Revised: Zhang et al., 2018; Karras et al.,

Original: [Figure]
Revised: [Figure]

Original: X B
Revised: X B

Original: Y B
Revised: Y B

Original: I AA
Revised: I AA

Original: I BB
Revised: I BB

Original: X A ………..
Revised: X A ………..

Original: I AB (intra
Revised: I AB (intra

Original: [Figure]
Revised: [Figure]

Original: ) (inter-domain) (intra-domain)
Revised: ) (inter-domain) (intra-domain)

Original: I I
Revised: I I

Original: I I
Revised: I I

Original: [Figure]
Revised: [Figure]

Original: ; Brock et al., 2019).
Revised: ; Brock et al., 2019).

Original: The premise behind learning disentangled representations is that an image can be thought of as a function of, say two independent latent factors, such that each controls only one human interpretable property (e.g., shape vs. appearance).
Revised: The premise behind learning disentangled representations is that an image can be thought of as a function of, say two independent latent factors, such that each controls only one human interpretable property (e.g., shape vs. appearance).

Original: The existence of such representations enables combining latent factors from two different source images to create a new one, which has properties of both.
Revised: The existence of such representations enables combining latent factors from two different source images to create a new one, which has properties of both.

Original: Prior generative modeling work (Hu et al., 2018; Singh et al., 2019; Li et al., 2020) explore a part of this idea, where the space of latent factors being combined is limited to one domain (e.g., combining a sparrow’s appearance with a duck’s shape within the domain of birds; I AA in Fig. 2), a scenario which we refer to as intra-domain disentanglement of latent factors.
Revised: Prior generative modeling work (Hu et al., 2018; Singh et al., 2019; Li et al., 2020) explore a part of this idea, where the space of latent factors being combined is limited to one domain (e.g., combining a sparrow’s appearance with a duck’s shape within the domain of birds; I AA in Fig. 2), a scenario which we refer to as intra-domain disentanglement of latent factors.

Original: This work, focusing on shape and appearance as factors, generalizes this idea to inter-domain disentanglement: combining latent factors from different domains (e.g., appearance from birds, shape from cars) to create a new breed of images which does not exist in either domain ( I AB in Fig. 2).
Revised: This work, focusing on shape and appearance as factors, generalizes this idea to inter-domain disentanglement: combining latent factors from different domains (e.g., appearance from birds, shape from cars) to create a new breed of images which does not exist in either domain ( I AB in Fig. 2).

Original: The key challenge to this problem is that there is no ground-truth distribution for the hybrid visual concept that spans the two domains.
Revised: The key challenge to this problem is that there is no ground-truth distribution for the hybrid visual concept that spans the two domains.

Original: Due to this, directly applying a single domain disentangled image generation approach to the multi-domain setting does not work, as the hybrid concept would be considered out of distribution (we provide more analysis in Sec. 3).
Revised: Due to this, directly applying a single domain disentangled image generation approach to the multi-domain setting does not work, as the hybrid concept would be considered out of distribution (we provide more analysis in Sec. 3).

Original: Despite the lack of ground-truth, as humans, we would deem certain combinations of factors to be better than others.
Revised: Despite the lack of ground-truth, as humans, we would deem certain combinations of factors to be better than others.

Original: For example, if two domains share object parts (e.g., dog and leopard), we would prefer a transfer of appearance in which local part appearances are preserved.
Revised: For example, if two domains share object parts (e.g., dog and leopard), we would prefer a transfer of appearance in which local part appearances are preserved.

Original: For the ones that don’t share object parts (e.g., bird and car), we may prefer a transfer of appearance in which the overall color/texture frequency is preserved (e.g. Fig.
Revised: For the ones that don’t share object parts (e.g., bird and car), we may prefer a transfer of appearance in which the overall color/texture frequency is preserved (e.g. Fig.

Original: 2, I 2 and I AB ), which has been found to be useful in object categorization at coarse level in neuroimaging study (Rice et al., 2014).
Revised: 2, I 2 and I AB ), which has been found to be useful in object categorization at coarse level in neuroimaging study (Rice et al., 2014).

Original: Our work formulates this idea as a training process, where any two images having the same latent appearance are constrained to have similar frequency of those low-level features.
Revised: Our work formulates this idea as a training process, where any two images having the same latent appearance are constrained to have similar frequency of those low-level features.

Original: These features in turn are learned (as opposed to being hand-crafted), using contrastive learning (Hadsell et al., 2006; Chen et al., 2020), to better capture the low-level statistics of the dataset.
Revised: These features in turn are learned (as opposed to being hand-crafted), using contrastive learning (Hadsell et al., 2006; Chen et al., 2020), to better capture the low-level statistics of the dataset.

Original: The net effect is an accurate transfer of appearance, where important details remain consistent across domains in spite of large shape changes.
Revised: The net effect is an accurate transfer of appearance, where important details remain consistent across domains in spite of large shape changes.

Original: Importantly, we achieve this by only requiring bounding box annotations to help disentangle object from background, without any other labels, including which domain an image comes from.
Revised: Importantly, we achieve this by only requiring bounding box annotations to help disentangle object from background, without any other labels, including which domain an image comes from.

Original: To our knowledge, our work is the ﬁrst to attempt combining factors from different data distributions to generate abstract visual concepts (e.g., car with dog’s texture).
Revised: To our knowledge, our work is the ﬁrst to attempt combining factors from different data distributions to generate abstract visual concepts (e.g., car with dog’s texture).

Original: We perform experiments on a variety of multi-modal datasets, and demonstrate our method’s effectiveness qualitatively, quantitatively, and through user studies.
Revised: We perform experiments on a variety of multi-modal datasets, and demonstrate our method’s effectiveness qualitatively, quantitatively, and through user studies.

Original: We believe our work can open up new avenues for art/design; e.g., a customer could visualize how sofas would look with an animal print or a fashion/car designer could create a new space of designs using the appearance from arbitrary objects.
Revised: We believe our work can open up new avenues for art/design; e.g., a customer could visualize how sofas would look with an animal print or a fashion/car designer could create a new space of designs using the appearance from arbitrary objects.

Original: Finally, we believe that the task introduced in this work offers better scrutiny of the quality of disentanglement learned by a method: if it succeeds in doing so within a domain but not in the presence of multiple ones, that in essence indicates some form of entanglement of factors with the domain’s properties.
Revised: Finally, we believe that the task introduced in this work offers better scrutiny of the quality of disentanglement learned by a method: if it succeeds in doing so within a domain but not in the presence of multiple ones, that in essence indicates some form of entanglement of factors with the domain’s properties.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Learning disentangled representations for image generation has been studied in both the supervised (relying on e.g., keypoints and object masks)
Revised: Learning disentangled representations for image generation has been studied in both the supervised (relying on e.g., keypoints and object masks)

Original: (Peng et al., 2017; Balakrishnan et al., 2018; Ma et al., 2018) and unsupervised settings (Li et al., 2018; Shu et al., 2018).
Revised: (Peng et al., 2017; Balakrishnan et al., 2018; Ma et al., 2018) and unsupervised settings (Li et al., 2018; Shu et al., 2018).

Original: Recent work disentangle object shape, appearance, pose, and background with only bounding box annotations (Singh et al., 2019;
Revised: Recent work disentangle object shape, appearance, pose, and background with only bounding box annotations (Singh et al., 2019;

Original: Li et al., 2020).
Revised: Li et al., 2020).

Original: All prior work, however, focus on disentangling and combining factors within a single domain (e.g., birds), and cannot be directly extended to the multi-domain setting since hybrid images would be considered out of distribution (i.e., fake).
Revised: All prior work, however, focus on disentangling and combining factors within a single domain (e.g., birds), and cannot be directly extended to the multi-domain setting since hybrid images would be considered out of distribution (i.e., fake).

Original: We present a framework that addresses this limitation, and which works equally well in both single and multi-domain settings.
Revised: We present a framework that addresses this limitation, and which works equally well in both single and multi-domain settings.

Original: Another potential angle to tackle the task at hand is through style-content disentanglement (Gatys et al., 2015; Johnson et al., 2016; Ulyanov et al., 2016).
Revised: Another potential angle to tackle the task at hand is through style-content disentanglement (Gatys et al., 2015; Johnson et al., 2016; Ulyanov et al., 2016).

Original: However, an object’s appearance and shape in complex datasets do not necessarily align with those of style and content (e.g., color of background dominating the style rather than object’s appearance).
Revised: However, an object’s appearance and shape in complex datasets do not necessarily align with those of style and content (e.g., color of background dominating the style rather than object’s appearance).

Original: Unsupervised image-to-image translation works (Zhu et al., 2017;
Revised: Unsupervised image-to-image translation works (Zhu et al., 2017;

Original: Kim et al., 2017; Huang et al., 2018; Gonzalez-Garcia et al., 2018; Choi et al., 2020) translate an image from domain A to domain B , such that the resulting image preserves the property common to domains A and B (e.g., structure), and property exclusive to B (e.g., appearance/style).
Revised: Kim et al., 2017; Huang et al., 2018; Gonzalez-Garcia et al., 2018; Choi et al., 2020) translate an image from domain A to domain B , such that the resulting image preserves the property common to domains A and B (e.g., structure), and property exclusive to B (e.g., appearance/style).

Original: However, if the domains don’t have anything in common (e.g., cars ↔ dogs : different structure and appearance), the translated images typically become degenerate, and no longer preserve properties from different domains.
Revised: However, if the domains don’t have anything in common (e.g., cars ↔ dogs : different structure and appearance), the translated images typically become degenerate, and no longer preserve properties from different domains.

Original: In contrast, our method can combine latent factors across arbitrary domains that have no part-level correspondences.
Revised: In contrast, our method can combine latent factors across arbitrary domains that have no part-level correspondences.

Original: Moreover, when part-level correspondences do exist (e.g., dogs ↔ tiger ), it combines appearance and shape in a way which preserves them.
Revised: Moreover, when part-level correspondences do exist (e.g., dogs ↔ tiger ), it combines appearance and shape in a way which preserves them.

Original: Lee et al.
Revised: Lee et al.

Original: (2018) extended the multimodal image-to-image translation setting by conditioning the translation process on both a content image as well as a query attribute image, so that the resulting output preserves the content and attributes of the respective images.
Revised: (2018) extended the multimodal image-to-image translation setting by conditioning the translation process on both a content image as well as a query attribute image, so that the resulting output preserves the content and attributes of the respective images.

Original: However, this application was explored in settings where both the content and attribute image share similar content/structure (e.g., natural and sketch images of face domain as content/attribute images respectively), which is different than our setting where factors to be combined come from entirely different domains (e.g., cars vs birds).
Revised: However, this application was explored in settings where both the content and attribute image share similar content/structure (e.g., natural and sketch images of face domain as content/attribute images respectively), which is different than our setting where factors to be combined come from entirely different domains (e.g., cars vs birds).

Original:  A PPROACH
Revised:  A PPROACH

Original: Given a single dataset consisting of two image domains A and B (e.g., dogs and cars), our goal is to learn a generative model of this distribution (with only bounding box annotations and without domain/category/segmentation labels), so that one latent factor (shape) from A and another factor (appearance) from B (or vice-versa) can be combined to generate a new out-of-distribution image preserving the respective latent properties from the two domains.
Revised: Given a single dataset consisting of two image domains A and B (e.g., dogs and cars), our goal is to learn a generative model of this distribution (with only bounding box annotations and without domain/category/segmentation labels), so that one latent factor (shape) from A and another factor (appearance) from B (or vice-versa) can be combined to generate a new out-of-distribution image preserving the respective latent properties from the two domains.

Original: Since there is no ground truth for the desired hybrid out-of-distribution images, our key idea is to preserve the frequency of low-level appearance features when transferred from one domain’s shape to another domain’s shape.
Revised: Since there is no ground truth for the desired hybrid out-of-distribution images, our key idea is to preserve the frequency of low-level appearance features when transferred from one domain’s shape to another domain’s shape.

Original: To this end, we develop a learnable, differentiable histogram-based representation of object appearance, and optimize the generator so that any two images that are assigned the same latent appearance factor produce similar feature histograms.
Revised: To this end, we develop a learnable, differentiable histogram-based representation of object appearance, and optimize the generator so that any two images that are assigned the same latent appearance factor produce similar feature histograms.

Original: This leads to the model learning better disentanglement of object shape and appearance, allowing it to create hybrid images that span multiple domains.
Revised: This leads to the model learning better disentanglement of object shape and appearance, allowing it to create hybrid images that span multiple domains.

Original: We ﬁrst formalize the desired properties of our model, and then discuss a single domain disentangled image generation base model (Singh et al., 2019) that we build upon.
Revised: We ﬁrst formalize the desired properties of our model, and then discuss a single domain disentangled image generation base model (Singh et al., 2019) that we build upon.

Original: Finally, we explain how our proposed framework can augment the base model to achieve the complete set of desired properties.
Revised: Finally, we explain how our proposed framework can augment the base model to achieve the complete set of desired properties.

Original: 3. P ROBLEM FORMULATION
Revised: 3. P ROBLEM FORMULATION

Original: Combining factors from multiple domains requires learning a disentangled latent representation for appearance, shape, and background of an object.
Revised: Combining factors from multiple domains requires learning a disentangled latent representation for appearance, shape, and background of an object.

Original: This enables each of the latent factor’s behavior to remain consistent, irrespective of other factors.
Revised: This enables each of the latent factor’s behavior to remain consistent, irrespective of other factors.

Original: For example, we want the latent appearance vector that represents red and blue object color to produce the same appearance (color/texture distribution) regardless of whether it is combined with a bird’s shape or a car’s shape ( I AA and I AB in Fig. 2).
Revised: For example, we want the latent appearance vector that represents red and blue object color to produce the same appearance (color/texture distribution) regardless of whether it is combined with a bird’s shape or a car’s shape ( I AA and I AB in Fig. 2).

Original: Henceforth, we represent shape, appearance, and background of an object as x , y , b respectively, and remaining continuous factors (e.g., object pose) using z .
Revised: Henceforth, we represent shape, appearance, and background of an object as x , y , b respectively, and remaining continuous factors (e.g., object pose) using z .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, we can interpret domain A as having an associated set of shapes - X A (e.g., possible bird shapes), and an associated set of appearances - Y A (e.g., possible bird appear- ances).
Revised: 2, we can interpret domain A as having an associated set of shapes - X A (e.g., possible bird shapes), and an associated set of appearances - Y A (e.g., possible bird appear- ances).

Original: With an analogous interpretation for domain B (e.g., cars), we formalize the following problems: (i) Intra-domain disentanglement: where a method can generate images of the form
Revised: With an analogous interpretation for domain B (e.g., cars), we formalize the following problems: (i) Intra-domain disentanglement: where a method can generate images of the form

Original: I =
Revised: I =

Original: G ( x, y, z, b ) , where [ x, y ] ∈ ( X A × Y A ) ( × denotes Cartesian product).
Revised: G ( x, y, z, b ) , where [ x, y ] ∈ ( X A × Y A ) ( × denotes Cartesian product).

Original: In other words, the task is to combine all possible shapes with possible appearances within a domain (e.g., I AA / I BB in Fig. 2).
Revised: In other words, the task is to combine all possible shapes with possible appearances within a domain (e.g., I AA / I BB in Fig. 2).

Original: (ii) Inter-domain disentanglement: where everything remains the same as intradomain disentanglement, except that [ x, y ] ∈ ( X AB × Y AB ) , where X AB =
Revised: (ii) Inter-domain disentanglement: where everything remains the same as intradomain disentanglement, except that [ x, y ] ∈ ( X AB × Y AB ) , where X AB =

Original: ( X A ∪ X B ) and
Revised: ( X A ∪ X B ) and

Original: Y AB =
Revised: Y AB =

Original: ( Y A ∪ Y B ) .
Revised: ( Y A ∪ Y B ) .

Original: This is a more general version of the task, where we wish to combine all possible shapes with all possible appearances across multiple domains (e.g., I AB in Fig. 2).
Revised: This is a more general version of the task, where we wish to combine all possible shapes with all possible appearances across multiple domains (e.g., I AB in Fig. 2).

Original: 3. B ASE MODEL
Revised: 3. B ASE MODEL

Original: We start with an intra-domain disentangled image generation approach, FineGAN (Singh et al., 2019), as our base model.
Revised: We start with an intra-domain disentangled image generation approach, FineGAN (Singh et al., 2019), as our base model.

Original: Fig.
Revised: Fig.

Original: 3 (left) provides a high level overview of its architecture, which consists of three image-generation stages: background, shape, and appearance.
Revised: 3 (left) provides a high level overview of its architecture, which consists of three image-generation stages: background, shape, and appearance.

Original: As a whole, FineGAN takes in four latent variables as input: a continuous noise vector z ∼ N (0 , 1) , and one-hot vectors x ∼ Cat ( K = N x , p = 1 /N x ) , y ∼ Cat ( K = N y , p = 1 /N y ) and b ∼ Cat ( K = N b , p = 1 /N b ) , where N x , N y , and N b are hyperparameters that represent the number of distinct shapes, appearances, and background to be discovered, respectively.
Revised: As a whole, FineGAN takes in four latent variables as input: a continuous noise vector z ∼ N (0 , 1) , and one-hot vectors x ∼ Cat ( K = N x , p = 1 /N x ) , y ∼ Cat ( K = N y , p = 1 /N y ) and b ∼ Cat ( K = N b , p = 1 /N b ) , where N x , N y , and N b are hyperparameters that represent the number of distinct shapes, appearances, and background to be discovered, respectively.

Original: Using these, it generates an image in a stagewise manner: (i) background stage generates a background image ( I b ); (ii) shape stage creates a shape image ( I x ) by drawing the silhouette of the object over the background using a shape mask ( m x ); (iii) the appearance stage generates the color/texture details, and stitches it to I x using an appearance mask ( m y ) to create the ﬁnal image ( I ).
Revised: Using these, it generates an image in a stagewise manner: (i) background stage generates a background image ( I b ); (ii) shape stage creates a shape image ( I x ) by drawing the silhouette of the object over the background using a shape mask ( m x ); (iii) the appearance stage generates the color/texture details, and stitches it to I x using an appearance mask ( m y ) to create the ﬁnal image ( I ).

Original: To learn to disentangle object from background, FineGAN relies on bounding box supervision.
Revised: To learn to disentangle object from background, FineGAN relies on bounding box supervision.

Original: To learn the desired disentanglement between shape and appearance without any supervision, FineGAN uses information theory to associate each factor to a latent code, and constrains the relationship between the codes to be hierarchical.
Revised: To learn the desired disentanglement between shape and appearance without any supervision, FineGAN uses information theory to associate each factor to a latent code, and constrains the relationship between the codes to be hierarchical.

Original: Speciﬁcally, it makes the assumption that N x < N y ; i.e., variety in object shape is less than variety in object appearance, which is generally true for realworld objects (e.g., different duck species have different colors/textures but share the same overall shape).
Revised: Speciﬁcally, it makes the assumption that N x < N y ; i.e., variety in object shape is less than variety in object appearance, which is generally true for realworld objects (e.g., different duck species have different colors/textures but share the same overall shape).

Original: During training, this hierarchy constraint is imposed so that a set of latent appearance codes y will always be paired with a particular latent shape code x .
Revised: During training, this hierarchy constraint is imposed so that a set of latent appearance codes y will always be paired with a particular latent shape code x .

Original: This ensures that, during training, the shape code representing e.g., a duck’s shape does not get paired with the appearance code for e.g., a sparrow’s texture to create an unnatural combination (which the discriminator could use to easily classify the generated image as fake, and in turn prevent the model from learning the desired disentanglement).
Revised: This ensures that, during training, the shape code representing e.g., a duck’s shape does not get paired with the appearance code for e.g., a sparrow’s texture to create an unnatural combination (which the discriminator could use to easily classify the generated image as fake, and in turn prevent the model from learning the desired disentanglement).

Original: Two types of loss functions are used.
Revised: Two types of loss functions are used.

Original: An adversarial loss L adv (Goodfellow et al., 2014) to enforce realism on the background
Revised: An adversarial loss L adv (Goodfellow et al., 2014) to enforce realism on the background

Original: I b (on a patch-level) and ﬁnal generated image I (on an image-level), and a mutual information loss L info (Chen et al., 2016) between (i) x and masked object shape ( m x ∗ I x ), (ii) y and masked object appearance ( m y ∗
Revised: I b (on a patch-level) and ﬁnal generated image I (on an image-level), and a mutual information loss L info (Chen et al., 2016) between (i) x and masked object shape ( m x ∗ I x ), (ii) y and masked object appearance ( m y ∗

Original: I ), to help the latent codes gain control over the respective factors:
Revised: I ), to help the latent codes gain control over the respective factors:

Original: [Equation]
Revised: [Equation]

Original: where c represents the input latent codes, r is the distribution of real patches/images, G/D are the generator and discriminator, G ( c ) is the generated image, and m is the generated mask.
Revised: where c represents the input latent codes, r is the distribution of real patches/images, G/D are the generator and discriminator, G ( c ) is the generated image, and m is the generated mask.

Original: For the background stage, only L adv is used on a patch-level, where r is the distribution of real patches and c = { z, b } .
Revised: For the background stage, only L adv is used on a patch-level, where r is the distribution of real patches and c = { z, b } .

Original: For the shape stage, only L info is used, with c = x and m = m x .
Revised: For the shape stage, only L info is used, with c = x and m = m x .

Original: Finally, for the appearance stage, both L adv and L info are used, where r is the distribution of real images and c = { z, b, x, y } for L adv , and c = y and m = m y for L info .
Revised: Finally, for the appearance stage, both L adv and L info are used, where r is the distribution of real images and c = { z, b, x, y } for L adv , and c = y and m = m y for L info .

Original: We denote all the losses used by the base model (FineGAN in this case) as L base .
Revised: We denote all the losses used by the base model (FineGAN in this case) as L base .

Original: Fig.
Revised: Fig.

Original: 3 (bottom) summarizes the properties learned by the base model.
Revised: 3 (bottom) summarizes the properties learned by the base model.

Original: 3. C OMBINING FACTORS FROM MULTIPLE DOMAINS
Revised: 3. C OMBINING FACTORS FROM MULTIPLE DOMAINS

Original: Although the base model can combine shape and appearance within a single domain (Fig. 2 bottom left), it has trouble doing so across different ones, as illustrated in Fig.
Revised: Although the base model can combine shape and appearance within a single domain (Fig. 2 bottom left), it has trouble doing so across different ones, as illustrated in Fig.

Original: 4 (b) top, where the same latent appearance code behaves differently when combined with a car’s shape ( I ) versus a bird’s shape ( I pos 2 ).
Revised: 4 (b) top, where the same latent appearance code behaves differently when combined with a car’s shape ( I ) versus a bird’s shape ( I pos 2 ).

Original: The main reason is because hybrid images, which combine e.g., a bird’s colors with a car’s shape, are non-existent in the real data (i.e., out of distribution).
Revised: The main reason is because hybrid images, which combine e.g., a bird’s colors with a car’s shape, are non-existent in the real data (i.e., out of distribution).

Original: Thus, the model is penalized from
Revised: Thus, the model is penalized from

Original: [Figure]
Revised: [Figure]

Original: learning such a combination via the adversarial loss.
Revised: learning such a combination via the adversarial loss.

Original: For creating hybrid objects within a domain (e.g., combining red sparrow color and seagull shape to create a non-existent red seagull), this is less of a problem, since the base model implicitly learns part semantics as it has to consistently colorize the correct part of an object regardless of its pose (i.e., it learns a pose-equivariant representation of appearance; see Fig.
Revised: For creating hybrid objects within a domain (e.g., combining red sparrow color and seagull shape to create a non-existent red seagull), this is less of a problem, since the base model implicitly learns part semantics as it has to consistently colorize the correct part of an object regardless of its pose (i.e., it learns a pose-equivariant representation of appearance; see Fig.

Original: 3 top-left).
Revised: 3 top-left).

Original: And since intra-domain objects share part semantics, during test time (in which the adversarial loss no longer plays a role), transferring e.g., a sparrow’s appearance to a seagull’s shape can still be achieved with some degree of success.
Revised: And since intra-domain objects share part semantics, during test time (in which the adversarial loss no longer plays a role), transferring e.g., a sparrow’s appearance to a seagull’s shape can still be achieved with some degree of success.

Original: However, when the two domains do not share any part semantics (e.g., birds and cars), the model would not know where e.g., the color of a bird’s head should go in a car.
Revised: However, when the two domains do not share any part semantics (e.g., birds and cars), the model would not know where e.g., the color of a bird’s head should go in a car.

Original: In other words, when trained in a multi-domain setting, the base model is unable to fully disentangle shape from appearance; i.e., there is still some form of entanglement of shape and appearance tied to each domain.
Revised: In other words, when trained in a multi-domain setting, the base model is unable to fully disentangle shape from appearance; i.e., there is still some form of entanglement of shape and appearance tied to each domain.

Original: So, how can we learn better disentanglement between appearance and shape in the multi-domain setting, without any ground truth for hybrid images?
Revised: So, how can we learn better disentanglement between appearance and shape in the multi-domain setting, without any ground truth for hybrid images?

Original: Our idea is to preserve the frequency of lowlevel visual concepts (color/texture) as a heuristic when transferring the appearance from one domain to another, and not transfer any shape information in the process (e.g. transferring car’s wheels to a dog).
Revised: Our idea is to preserve the frequency of low-level visual concepts (color/texture) as a heuristic when transferring the appearance from one domain to another, and not transfer any shape information in the process (e.g. transfer a car’s color and texture distribution to a dog, while preventing transfer of its wheels).

Original: Speciﬁcally, any two generated images with the same latent appearance code (but different shape codes) should have similar frequency in low-level appearance.
Revised: Speciﬁcally, any two generated images with the same latent appearance code (but different shape codes) should have similar frequency in low-level appearance.

Original: To use this idea as part of an optimization process, we need to: (i) deﬁne the concepts whose frequency needs to be computed and make its computation differentiable, and (ii) induce the generator to generate hybrid images that preserve the low-level appearance frequency.
Revised: To use this idea as part of an optimization process, we need to: (i) deﬁne the concepts whose frequency needs to be computed and make its computation differentiable, and (ii) induce the generator to generate hybrid images that preserve the low-level appearance frequency.

Original: Learning a differentiable histogram of low-level features.
Revised: Learning a differentiable histogram of low-level features.

Original: At a high-level we’d like texture and color frequency to be our low-level feature concepts.
Revised: At a high-level we’d like texture and color frequency to be our low-level feature concepts.

Original: However, since it is difﬁcult to know a priori which speciﬁc textures/colors would be useful (e.g., a feature representing leopard texture would be useful for shape/appearance disentanglement in animals ↔ cars , but might not be for birds ↔ cars ), we propose to learn them in a data driven way.
Revised: However, since it is difﬁcult to know a priori which speciﬁc textures/colors would be useful (e.g., a feature representing leopard texture would be useful for shape/appearance disentanglement in animals ↔ cars , but might not be for birds ↔ cars ), we propose to learn them in a data driven way.

Original: Speciﬁcally, we represent the lowlevel concepts using a set of learnable convolutional ﬁlters (Fig. 4(a)), each of which convolves over a generated image to give regions of high correlation.
Revised: Speciﬁcally, we represent the lowlevel concepts using a set of learnable convolutional ﬁlters (Fig. 4(a)), each of which convolves over a generated image to give regions of high correlation.

Original: To focus on the object region, we mask out the background using the base model’s generated foreground mask, and compute the channelwise sum of the resulting response maps to get a k -dimensional histogram representation h .
Revised: To focus on the object region, we mask out the background using the base model’s generated foreground mask, and compute the channelwise sum of the resulting response maps to get a k -dimensional histogram representation h .

Original: This representation approximates the frequency of visual concepts represented by the set of ﬁlters (e.g., dog’s fur, tiger’s texture).
Revised: This representation approximates the frequency of visual concepts represented by the set of ﬁlters (e.g., dog’s fur, tiger’s texture).

Original: How do we ensure that the histogram h being approximated is of some relevant object feature?
Revised: How do we ensure that the histogram h being approximated is of some relevant object feature?

Original: The information captured by the ﬁlter bank should be such that (i) the same object in different pose/viewpoints has a similar appearance representation, (ii) which in turn should be dissimilar than
Revised: The information captured by the ﬁlter bank should be such that (i) the same object in different pose/viewpoints has a similar appearance representation, (ii) which in turn should be dissimilar than

Original: [Figure]
Revised: [Figure]

Original: the representation of any other arbitrary object.
Revised: the representation of any other arbitrary object.

Original: We hence train the ﬁlter bank using contrastive learning (Bachman et al., 2019; Chen et al., 2020).
Revised: We hence train the ﬁlter bank using contrastive learning (Bachman et al., 2019; Chen et al., 2020).

Original: Speciﬁcally, we ﬁrst construct a batch of N generated images, where each image is of the form I = G ( x 1 , y 1 , z 1 , b 1 ) .
Revised: Speciﬁcally, we ﬁrst construct a batch of N generated images, where each image is of the form I = G ( x 1 , y 1 , z 1 , b 1 ) .

Original: We then construct another batch, where each image is of the form I pos 

Revised: We then construct another batch, where each image is of the form I pos

Original: = G ( x 1 , y 1 , z 2 , b 1 ) , by only altering the noise vector z , which changes only the object’s pose (due to our base model; Fig.
Revised: = G ( x 1 , y 1 , z 2 , b 1 ) , by only altering the noise vector z , which changes only the object’s pose (due to our base model; Fig.

Original: 3 top-left).
Revised: 3 top-left).

Original: We set each I and corresponding I pos 1 pair as positive samples, and all other image pairs as negative samples (Fig. 4(b)).
Revised: We set each I and corresponding I pos 1 pair as positive samples, and all other image pairs as negative samples (Fig. 4(b)).

Original: Finally, we use NT - Xent
Revised: Finally, we use NT - Xent

Original: (Normalized temperature-scaled cross entropy loss (Chen et al., 2020)) for each image I in the batch to learn the ﬁlter bank:
Revised: (Normalized temperature-scaled cross entropy loss (Chen et al., 2020)) for each image I in the batch to learn the ﬁlter bank:

Original: [Equation]
Revised: [Equation]

Original: where h i represents the feature histogram of i th image, j and k indexes the positive and negative samples for i , respectively, sim denotes cosine similarity, and τ is the temperature hyperparameter (set to 0 . 5 ).
Revised: where h i represents the feature histogram of i th image, j and k indexes the positive and negative samples for i , respectively, sim denotes cosine similarity, and τ is the temperature hyperparameter (set to 0 . 5 ).

Original: We refer to the whole loss as L filter =
Revised: We refer to the whole loss as L filter =

Original: (cid:80) Ni =1 (cid:96) i , and optimize it by only updating the weights of the ﬁlter bank.
Revised: (cid:80) Ni =1 (cid:96) i , and optimize it by only updating the weights of the ﬁlter bank.

Original: Conditioning the generator to generate hybrid images.
Revised: Conditioning the generator to generate hybrid images.

Original: Now that we have a way to learn meaningful low-level features in a data driven way, we can use them to condition the generator to generate hybrid images.
Revised: Now that we have a way to learn meaningful low-level features in a data driven way, we can use them to condition the generator to generate hybrid images.

Original: We use an objective similar to L filter , where we use the same I = G ( x 1 , y 1 , z 1 , b 1 ) .
Revised: We use an objective similar to L filter , where we use the same I = G ( x 1 , y 1 , z 1 , b 1 ) .

Original: However, we construct the positive pairs differently:
Revised: However, we construct the positive pairs differently:

Original: I pos 2 =
Revised: I pos 2 =

Original: G ( x 2 , y 1 , z 1 , b 1 ) , in which the latent shape code x is different than that of I (while all other codes are the same); see Fig. 4(b).
Revised: G ( x 2 , y 1 , z 1 , b 1 ) , in which the latent shape code x is different than that of I (while all other codes are the same); see Fig. 4(b).

Original: We refer to this loss as L hybrid =
Revised: We refer to this loss as L hybrid =

Original: (cid:80) Ni =1 (cid:96) i , and optimize only the generator G ’s parameters.
Revised: (cid:80) Ni =1 (cid:96) i , and optimize only the generator G ’s parameters.

Original: Intuitively,
Revised: Intuitively,

Original: L hybrid pushes the generator to match the frequency of low-level appearance features (e.g., car’s color) between the generated images of two arbitrary shaped objects (e.g., car and bird).
Revised: L hybrid pushes the generator to match the frequency of low-level appearance features (e.g., car’s color) between the generated images of two arbitrary shaped objects (e.g., car and bird).

Original: The overall loss function of our framework is:
Revised: The overall loss function of our framework is:

Original: [Equation]
Revised: [Equation]

Original: where L base induces the properties needed for intra-domain disentanglement, and the combination of L filter and L hybrid helps extend the abilities to learn inter-domain disentanglement of factors.
Revised: where L base induces the properties needed for intra-domain disentanglement, and the combination of L filter and L hybrid helps extend the abilities to learn inter-domain disentanglement of factors.

Original: Note that we do not employ the adversarial loss (which is part of L base ) on any hybrid images that do not follow our base model’s hierarchical code constraints (as described in Sec. 3.2) since those images would be outside the real data distribution.
Revised: Note that we do not employ the adversarial loss (which is part of L base ) on any hybrid images that do not follow our base model’s hierarchical code constraints (as described in Sec. 3.2) since those images would be outside the real data distribution.

Original:  R ESULTS
Revised:  R ESULTS

Original: We now compare the proposed framework against several baselines, and study how they fare in terms of disentangling and combining shape/appearance from different domains.
Revised: We now compare the proposed framework against several baselines, and study how they fare in terms of disentangling and combining shape/appearance from different domains.

Original: We use three single domain datasets: CUB (Wah et al., 2011) collection of 200 birds categories.
Revised: We use three single domain datasets: CUB (Wah et al., 2011) collection of 200 birds categories.

Original: (ii)
Revised: (ii)

Original: Stanford Dogs (Khosla et al., 2011) collection of 120 dog categories.
Revised: Stanford Dogs (Khosla et al., 2011) collection of 120 dog categories.

Original: (iii) Stanford Cars (Krause et al., 2013) collection of 196 car categories.
Revised: (iii) Stanford Cars (Krause et al., 2013) collection of 196 car categories.

Original: Using these, we construct three multi-domain settings:
Revised: Using these, we construct three multi-domain settings:

Original: [Figure]
Revised: [Figure]

Original: birds → cars cars → dogs dogs → birds
Revised: birds → cars cars → dogs dogs → birds

Original: Input
Revised: Input

Original: Translation
Revised: Translation

Original: (i) birds ↔ cars, (ii) cars ↔ dogs, (iii) dogs ↔ birds, consisting of a total of 396, 316, andobject categories, respectively.
Revised: (i) birds ↔ cars, (ii) cars ↔ dogs, (iii) dogs ↔ birds, consisting of a total of 396, 316, andobject categories, respectively.

Original: We also use a multi-domain dataset of animal faces (Liu et al., 2019), comprising 149 carnivorous categories, where each category can be interpreted as a different domain.
Revised: We also use a multi-domain dataset of animal faces (Liu et al., 2019), comprising 149 carnivorous categories, where each category can be interpreted as a different domain.

Original: This forms our fourth multi-domain setting, (iv) animals ↔ animals.
Revised: This forms our fourth multi-domain setting, (iv) animals ↔ animals.

Original: 4. C AN IMAGE TRANSLATION OR STYLE TRANSFER METHODS SOLVE THIS TASK ?
Revised: 4. C AN IMAGE TRANSLATION OR STYLE TRANSFER METHODS SOLVE THIS TASK ?

Original: As mentioned in Sec 2, there are existing works which could potentially offer solutions for the task at hand, even with their core objective being different.
Revised: As mentioned in Sec 2, there are existing works which could potentially offer solutions for the task at hand, even with their core objective being different.

Original: CycleGAN (Zhu et al., 2017) and
Revised: CycleGAN (Zhu et al., 2017) and

Original: MUNIT (Huang et al., 2018) translate images from one domain to another, in a way where the translated images preserve some properties from both the source and target.
Revised: MUNIT (Huang et al., 2018) translate images from one domain to another, in a way where the translated images preserve some properties from both the source and target.

Original: If the two domains are, say cars ↔ dogs, can these methods translate a car into an image that combines the car’s shape with a dog-like appearance?
Revised: If the two domains are, say cars ↔ dogs, can these methods translate a car into an image that combines the car’s shape with a dog-like appearance?

Original: Fig.
Revised: Fig.

Original: 5 shows some translation results: in general, we observe that the translations become degenerate, and don’t preserve any interpretable property from the input, i.e., the methods are no longer able to disentangle structure from domain-speciﬁc appearance.
Revised: 5 shows some translation results: in general, we observe that the translations become degenerate, and don’t preserve any interpretable property from the input, i.e., the methods are no longer able to disentangle structure from domain-speciﬁc appearance.

Original: We also explore AdaIn (Huang & Belongie, 2017), a method capable of transferring style between any arbitrary pair of images; see Fig.
Revised: We also explore AdaIn (Huang & Belongie, 2017), a method capable of transferring style between any arbitrary pair of images; see Fig.

Original: We notice that the deﬁnition of style transferred and what we perceive as appearance don’t always align, and even then, the method is unable to solely focus on the foreground object when extracting and transferring the style (e.g., bird’s blue color bleeds into the background of the car in the ﬁrst row).
Revised: We notice that the deﬁnition of style transferred and what we perceive as appearance don’t always align, and even then, the method is unable to solely focus on the foreground object when extracting and transferring the style (e.g., bird’s blue color bleeds into the background of the car in the ﬁrst row).

Original: Overall, Fig.
Revised: Overall, Fig.

Original: 5 shows that these methods are not suited for settings involving arbitrary domains, which have no similarity in object shape/appearance.
Revised: 5 shows that these methods are not suited for settings involving arbitrary domains, which have no similarity in object shape/appearance.

Original: 4. C OMPARISON TO MULTI - FACTOR DISENTANGLEMENT BASELINES
Revised: 4. C OMPARISON TO MULTI - FACTOR DISENTANGLEMENT BASELINES

Original: We next study baselines which disentangle object shape, appearance, pose, and background, and generate images conditioned on the respective latent vectors: (i) FineGAN (Singh et al., 2019): base model described in Sec. 3.2; (ii) Relaxed FineGAN : an extension of FineGAN, where we relax the hierarchical constraint between shape and appearance codes (so that at least conceptually, hybrid images can be generated during training).
Revised: We next study baselines which disentangle object shape, appearance, pose, and background, and generate images conditioned on the respective latent vectors: (i) FineGAN (Singh et al., 2019): base model described in Sec. 3.2; (ii) Relaxed FineGAN : an extension of FineGAN, where we relax the hierarchical constraint between shape and appearance codes (so that at least conceptually, hybrid images can be generated during training).

Original: For those hybrids, we only optimize L info whereas for the remaining images (generated with code constraints), we optimize the full L base .
Revised: For those hybrids, we only optimize L info whereas for the remaining images (generated with code constraints), we optimize the full L base .

Original: (iii)
Revised: (iii)

Original: Ours w/o L filter : our approach using a ﬁxed, randomly initialized ﬁlter bank to compute the frequencybased representation; (iv) Ours : our ﬁnal approach.
Revised: Ours w/o L filter : our approach using a ﬁxed, randomly initialized ﬁlter bank to compute the frequencybased representation; (iv) Ours : our ﬁnal approach.

Original: Note that we don’t need domain labels to train methods (i)-(iv); but once trained, we can visualize what the different learned shape ( x ) and appearance ( y ) latent vectors represent (e.g., x 1 and x 2 might generate a speciﬁc bird and car shape, respectively).
Revised: Note that we don’t need domain labels to train methods (i)-(iv); but once trained, we can visualize what the different learned shape ( x ) and appearance ( y ) latent vectors represent (e.g., x 1 and x 2 might generate a speciﬁc bird and car shape, respectively).

Original: Hence, for evaluation purposes, we manually create a split of the latent vectors, [ X A ,
Revised: Hence, for evaluation purposes, we manually create a split of the latent vectors, [ X A ,

Original: Y A ] and [ X B , Y B ], so that ∀ x ∈ X A , x represents only one domain (e.g., birds’ shapes).
Revised: Y A ] and [ X B , Y B ], so that ∀ x ∈ X A , x represents only one domain (e.g., birds’ shapes).

Original: This way, we can combine latent factors from different domains to study shape/appearance disentanglement.
Revised: This way, we can combine latent factors from different domains to study shape/appearance disentanglement.

Original: We ﬁrst show qualitative comparisons to FineGAN over different multi-domain datasets in Fig. 6.
Revised: We ﬁrst show qualitative comparisons to FineGAN over different multi-domain datasets in Fig. 6.

Original: FineGAN has issues combining factors from different domains, evident through (i) inaccurate ap-
Revised: FineGAN has issues combining factors from different domains, evident through (i) inaccurate ap-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: By enforcing similarity in the frequency of low-level features (via L filter + L hybrid ), our model better retains the color/texture distribution across the source and target compared to alternate baselines.
Revised: By enforcing similarity in the frequency of low-level features (via L filter + L hybrid ), our model better retains the color/texture distribution across the source and target compared to alternate baselines.

Original: pearance transfer (e.g., row 1: only car window color gets transferred to the bird), and (ii) the inability to transfer details beyond rough color (e.g., row 4: the fur texture of the brown dog isn’t transferred to the car; row 1: the characteristic leopard texture isn’t transferred to the dog).
Revised: pearance transfer (e.g., row 1: only car window color gets transferred to the bird), and (ii) the inability to transfer details beyond rough color (e.g., row 4: the fur texture of the brown dog isn’t transferred to the car; row 1: the characteristic leopard texture isn’t transferred to the dog).

Original: Our method, on the other hand, results in a more holistic transfer of appearance to arbitrary shapes.
Revised: Our method, on the other hand, results in a more holistic transfer of appearance to arbitrary shapes.

Original: Upon transferring, it is able to better preserve the (i) color distribution (row 5/6: both the blue and red components get transferred to the car/bird respectively), (ii) other low-level characteristic features (row 8: brown fur gets transferred to the car’s body; row 5: leopard’s texture is transferred to the dog’s face).
Revised: Upon transferring, it is able to better preserve the (i) color distribution (row 5/6: both the blue and red components get transferred to the car/bird respectively), (ii) other low-level characteristic features (row 8: brown fur gets transferred to the car’s body; row 5: leopard’s texture is transferred to the dog’s face).

Original: Focusing on animals ↔ animals results for FineGAN, we see that the Shape images have a tendency to resist appearance change (e.g., the leopard’s texture is still preserved in the output image), which indicates there is some entanglement between shape and appearance.
Revised: Focusing on animals ↔ animals results for FineGAN, we see that the Shape images have a tendency to resist appearance change (e.g., the leopard’s texture is still preserved in the output image), which indicates there is some entanglement between shape and appearance.

Original: This is mainly because unlike our approach, FineGAN does not have an explicit constraint to enforce the same latent appearance code to produce the same low-level visual features regardless of shape.
Revised: This is mainly because unlike our approach, FineGAN does not have an explicit constraint to enforce the same latent appearance code to produce the same low-level visual features regardless of shape.

Original: We study this property in more detail in Sec.
Revised: We study this property in more detail in Sec.

Original: A.3.
Revised: A.3.

Original: [Table]
Revised: [Table]

Original: 4.2. Q UANTIFYING SHAPE / APPEARANCE DISENTANGLEMENT
Revised: 4.2. Q UANTIFYING SHAPE / APPEARANCE DISENTANGLEMENT

Original: Appearance transfer: In this experiment, we evaluate how well appearance is transferred from one domain to another.
Revised: Appearance transfer: In this experiment, we evaluate how well appearance is transferred from one domain to another.

Original: For each latent appearance vector y i , we generate a source I s =
Revised: For each latent appearance vector y i , we generate a source I s =

Original: G ( x i , y i , b i , z i ) and target image I t =
Revised: G ( x i , y i , b i , z i ) and target image I t =

Original: G ( x j , y i , b i , z i ) ( x i , x j represent shapes from different domains).
Revised: G ( x j , y i , b i , z i ) ( x i , x j represent shapes from different domains).

Original: We segment the foreground pixels using DeepLabv3 (Chen et al., 2017), pre-trained on COCO (Lin et al., 2014).
Revised: We segment the foreground pixels using DeepLabv3 (Chen et al., 2017), pre-trained on COCO (Lin et al., 2014).

Original: We then evaluate the similarity between the source and target images’ foreground appearance by computing the χ 2 -distance between their color and texture histograms (Leung & Malik, 2001) (more details in Sec.
Revised: We then evaluate the similarity between the source and target images’ foreground appearance by computing the χ 2 -distance between their color and texture histograms (Leung & Malik, 2001) (more details in Sec.

Original: A.4.1).
Revised: A.4.1).

Original: Table 1 summarizes the results.
Revised: Table 1 summarizes the results.

Original: As expected based on Figs.
Revised: As expected based on Figs.

Original: 5 and 6, the baselines often have issues accurately transferring the color/texture distribution to an arbitrary shape, resulting in the color/texture histogram for I s
Revised: 5 and 6, the baselines often have issues accurately transferring the color/texture distribution to an arbitrary shape, resulting in the color/texture histogram for I s

Original: and I t having low similarity.
Revised: and I t having low similarity.

Original: ‘Ours w/o L filter ’ is a decent baseline, which simply uses random ﬁlters to minimize L hybrid .
Revised: ‘Ours w/o L filter ’ is a decent baseline, which simply uses random ﬁlters to minimize L hybrid .

Original: Our ﬁnal approach, which learns the ﬁlters, achieves the best overall performance in retaining the color and texture information.
Revised: Our ﬁnal approach, which learns the ﬁlters, achieves the best overall performance in retaining the color and texture information.

Original: Shape transfer: We next evaluate how well the object’s original shape is retained when appearance from another domain is transferred to it.
Revised: Shape transfer: We next evaluate how well the object’s original shape is retained when appearance from another domain is transferred to it.

Original: If shape is accurately disentangled from appearance, then if we generate a stack of images { G ( x, y i , z, b ) } ∨ y i ∈ S (arbitrary set of latent appearances), any differences should only be in the foreground appearance (Fig.
Revised: If shape is accurately disentangled from appearance, then if we generate a stack of images { G ( x, y i , z, b ) } ∨ y i ∈ S (arbitrary set of latent appearances), any differences should only be in the foreground appearance (Fig.

Original: 3 top right).
Revised: 3 top right).

Original: We therefore compute the standard deviation across the stack, and convert it into a binary mask using an appropriate threshold.
Revised: We therefore compute the standard deviation across the stack, and convert it into a binary mask using an appropriate threshold.

Original: If different sets of appearances ( S ) result in similar masks, that is an indication of shape being controlled only by x .
Revised: If different sets of appearances ( S ) result in similar masks, that is an indication of shape being controlled only by x .

Original: So, we randomly split the available latent vectors into two sets, S 1 and S 2 , compute their respective masks, and evaluate the IoU between them (Sec. A.4.2).
Revised: So, we randomly split the available latent vectors into two sets, S 1 and S 2 , compute their respective masks, and evaluate the IoU between them (Sec. A.4.2).

Original: This is repeated for 10 different random splits of S 1 / S 2 , giving the score for a latent shape.
Revised: This is repeated for 10 different random splits of S 1 / S 2 , giving the score for a latent shape.

Original: The ﬁnal score is averaged over all possible latent shapes.
Revised: The ﬁnal score is averaged over all possible latent shapes.

Original: Table 2 (left) summarizes the results: we see that both FineGAN and our method achieve high IoU scores, demonstrating that changes introduced by different appearance vectors are mostly conﬁned within the same region, and hence don’t change the object shape.
Revised: Table 2 (left) summarizes the results: we see that both FineGAN and our method achieve high IoU scores, demonstrating that changes introduced by different appearance vectors are mostly conﬁned within the same region, and hence don’t change the object shape.

Original: Hybrid images through the eyes of image classiﬁers: If we look at the last row in Fig.
Revised: Hybrid images through the eyes of image classiﬁers: If we look at the last row in Fig.

Original: 6, a regular car becomes a car with brown fur.
Revised: 6, a regular car becomes a car with brown fur.

Original: For an image classiﬁcation network, does this furry-car have more dog-like properties, and less of cars’?
Revised: For an image classiﬁcation network, does this furry-car have more dog-like properties, and less of cars’?

Original: When compared to the hybrids created by FineGAN, we ﬁnd our hybrids do indeed exhibit this behavior; see Sec A.5 in Appendix for details.
Revised: When compared to the hybrids created by FineGAN, we ﬁnd our hybrids do indeed exhibit this behavior; see Sec A.5 in Appendix for details.

Original: Perceptual study: Finally, we conduct human studies.
Revised: Perceptual study: Finally, we conduct human studies.

Original: Each task shows the results of two methods, one of which is ours and the other is either AdaIn or FineGAN.
Revised: Each task shows the results of two methods, one of which is ours and the other is either AdaIn or FineGAN.

Original: The result format is same as in Fig.
Revised: The result format is same as in Fig.

Original: 6, i.e., [Appearance , Shape → Output] (Appearance/Shape images are used as Style/Content input for AdaIn).
Revised: 6, i.e., [Appearance , Shape → Output] (Appearance/Shape images are used as Style/Content input for AdaIn).

Original: Amazon Mechanical Turkers are asked to select the method which resulted in a better shape/appearance transfer.
Revised: Amazon Mechanical Turkers are asked to select the method which resulted in a better shape/appearance transfer.

Original: We generate 250 tasks for each dataset, and gather responses from 5 unique turkers for each task, resulting in 1250 judgements.
Revised: We generate 250 tasks for each dataset, and gather responses from 5 unique turkers for each task, resulting in 1250 judgements.

Original: Table 2 (Right) summarizes the results.
Revised: Table 2 (Right) summarizes the results.

Original: Averaged across all datasets, our method gets chosen over AdaIn and FineGAN in 81.39 ± 3.85% and 74.38 ± 7.11% of the cases, demonstrating consistent superiority in generating hybrid images.
Revised: Averaged across all datasets, our method gets chosen over AdaIn and FineGAN in 81.39 ± 3.85% and 74.38 ± 7.11% of the cases, demonstrating consistent superiority in generating hybrid images.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: The method introduced in this work bridges the gap between learning intra and inter-domain disentanglement, and improves the base model through better appearance transfer between different domains.
Revised: The method introduced in this work bridges the gap between learning intra and inter-domain disentanglement, and improves the base model through better appearance transfer between different domains.

Original: However, there are some limitations: our method builds upon FineGAN, which in turn makes some assumptions about the data (existence of a hierarchy between shape and appearance factors) to learn the desired disentanglement.
Revised: However, there are some limitations: our method builds upon FineGAN, which in turn makes some assumptions about the data (existence of a hierarchy between shape and appearance factors) to learn the desired disentanglement.

Original: Consequently, our method is suited for datasets which possess these properties.
Revised: Consequently, our method is suited for datasets which possess these properties.

Original: Furthermore, when the number of domains increase, it becomes less feasible to train models on all combinations of domains ( (cid:0) n 2 (cid:1) , which grows quadratically).
Revised: Furthermore, when the number of domains increase, it becomes less feasible to train models on all combinations of domains ( (cid:0) n 2 (cid:1) , which grows quadratically).

Original: Another limitation is that our approach focuses on combining appearance and shape, but cannot combine different parts across domains (e.g., to create centaur).
Revised: Another limitation is that our approach focuses on combining appearance and shape, but cannot combine different parts across domains (e.g., to create centaur).

Original: Nonetheless, we believe that we address an important problem, which can also have practical applications in art and design.
Revised: Nonetheless, we believe that we address an important problem, which can also have practical applications in art and design.

Original: A A PPENDIX A. T RAINING DETAILS
Revised: A A PPENDIX A. T RAINING DETAILS

Original: Since we use FineGAN as our base model, the generator and discriminator architecture, as well as hyperparameters like learning rate, optimizer (Adam (Kingma & Welling, 2014)), β 1 / β 2 are borrowed from (Singh et al., 2019).
Revised: Since we use FineGAN as our base model, the generator and discriminator architecture, as well as hyperparameters like learning rate, optimizer (Adam (Kingma & Welling, 2014)), β 1 / β 2 are borrowed from (Singh et al., 2019).

Original: We also use the same data preprocessing step, where we crop the images to 1.5 × their available bounding box, so as to have a decent object to image size ratio.
Revised: We also use the same data preprocessing step, where we crop the images to 1.5 × their available bounding box, so as to have a decent object to image size ratio.

Original: We use a batch size of 24 to maximize the usage of two TITAN
Revised: We use a batch size of 24 to maximize the usage of two TITAN

Original: Xp GPUs per experiment.
Revised: Xp GPUs per experiment.

Original: Apart from the raw data, FineGAN uses two forms of information to disentangle object pose, shape, appearance and background while modeling an image distribution - (i) bounding box information, so that patches outside it can be used as real background patches to model background on a patch level; (ii)
Revised: Apart from the raw data, FineGAN uses two forms of information to disentangle object pose, shape, appearance and background while modeling an image distribution - (i) bounding box information, so that patches outside it can be used as real background patches to model background on a patch level; (ii)

Original: No. of object categories in the dataset.
Revised: No. of object categories in the dataset.

Original: The hierarchy to be discovered needs to be decided beforehand, where the number of distinct appearance ( N y ) are set to be the number of object categories, and the number of shapes ( N x ) is set empirically.
Revised: The hierarchy to be discovered needs to be decided beforehand, where the number of distinct appearance ( N y ) are set to be the number of object categories, and the number of shapes ( N x ) is set empirically.

Original: We set N y as 400 ( ∼ 200 + 196) for birds ↔ cars, 320 ( ∼ 196 + 120) for cars ↔ dogs and 320 (120 + 200) for dogs ↔ birds.
Revised: We set N y as 400 ( ∼ 200 + 196) for birds ↔ cars, 320 ( ∼ 196 + 120) for cars ↔ dogs and 320 (120 + 200) for dogs ↔ birds.

Original: We approximate thecar categories as 200, since it helps enforce the hierarchy constraints more easily.
Revised: We approximate thecar categories as 200, since it helps enforce the hierarchy constraints more easily.

Original: Furthermore, N x = N y /

Revised: Furthermore, N x = N y /

Original: for all these three datasets.
Revised: for all these three datasets.

Original: Note that this was the same conﬁguration used by FineGAN ( N x = N y /10), which the authors found to be a good default relationship to disentangle shape and appearance in multiple datasets.
Revised: Note that this was the same conﬁguration used by FineGAN ( N x = N y /10), which the authors found to be a good default relationship to disentangle shape and appearance in multiple datasets.

Original: The original paper also studies the sensitivity of the generated images’ quality to the choice of N x /N y , and found that the results are largely agnostic to these choices (except when those two values become very skewed: e.g. N x = 5).
Revised: The original paper also studies the sensitivity of the generated images’ quality to the choice of N x /N y , and found that the results are largely agnostic to these choices (except when those two values become very skewed: e.g. N x = 5).

Original: For animals ↔ animals, we set
Revised: For animals ↔ animals, we set

Original: N y as 150 ( ∼ 149 categories), and N x as N y /5.
Revised: N y as 150 ( ∼ 149 categories), and N x as N y /5.

Original: Upon training FineGAN, the four different latent vectors start to correspond to some properties in the generated images: latent appearance vector, having a one-hot representation, starts capturing the appearance (different indices start generating different appearance details) etc.
Revised: Upon training FineGAN, the four different latent vectors start to correspond to some properties in the generated images: latent appearance vector, having a one-hot representation, starts capturing the appearance (different indices start generating different appearance details) etc.

Original: This way, one can combine the latent shape code which generated duck, latent appearance code which generated yellow bird, to create a yellow duck.
Revised: This way, one can combine the latent shape code which generated duck, latent appearance code which generated yellow bird, to create a yellow duck.

Original: For birds ↔ cars, cars ↔ dogs, and dogs ↔ birds, we ﬁrst train the base model until convergence (600 epochs).
Revised: For birds ↔ cars, cars ↔ dogs, and dogs ↔ birds, we ﬁrst train the base model until convergence (600 epochs).

Original: This model can disentangle disentangle the four factors within a domain as shown in Fig. 3.
Revised: This model can disentangle disentangle the four factors within a domain as shown in Fig. 3.

Original: Starting from this pre-trained model, we now use our loss objectives L filter and L hybrid , in addition to L base (FineGAN’s overall objective), and further train the model to better perform inter-domain disentanglement of factors.
Revised: Starting from this pre-trained model, we now use our loss objectives L filter and L hybrid , in addition to L base (FineGAN’s overall objective), and further train the model to better perform inter-domain disentanglement of factors.

Original: Speciﬁcally, in each iteration, we optimize the model using L base , then optimize the generator’s weights using L hybrid , and then optimize the ﬁlters’ weights using L filter .
Revised: Speciﬁcally, in each iteration, we optimize the model using L base , then optimize the generator’s weights using L hybrid , and then optimize the ﬁlters’ weights using L filter .

Original: In our experiments, we found that using L hybrid at lesser frequency (1/4th of the time) compared to L base / L filter works better.
Revised: In our experiments, we found that using L hybrid at lesser frequency (1/4th of the time) compared to L base / L filter works better.

Original: The second phase of training is performed for about (50k iterations).
Revised: The second phase of training is performed for about (50k iterations).

Original: The two phase training for these three datasets is important, because we observed that directly using the combination of
Revised: The two phase training for these three datasets is important, because we observed that directly using the combination of

Original: L base , L hybrid and L filter while training from scratch leads to model having some issues with learning intra-domain disentanglement.
Revised: L base , L hybrid and L filter while training from scratch leads to model having some issues with learning intra-domain disentanglement.

Original: For the fourth dataset (animals ↔ animals), however, we directly train from scratch using all three loss components, a setting which actually results in better performance than the two phase one discussed above.
Revised: For the fourth dataset (animals ↔ animals), however, we directly train from scratch using all three loss components, a setting which actually results in better performance than the two phase one discussed above.

Original: This difference could be attributed to the difference in the nature of multi-domain datasets, where birds ↔ cars has only two domains, the animals ↔ animals domain effectively has N p (= 30) domains.
Revised: This difference could be attributed to the difference in the nature of multi-domain datasets, where birds ↔ cars has only two domains, the animals ↔ animals domain effectively has N p (= 30) domains.

Original: A. A BLATION STUDY
Revised: A. A BLATION STUDY

Original: We now study different design choices, and visualize their effects in mixing shape/appearance across domains.
Revised: We now study different design choices, and visualize their effects in mixing shape/appearance across domains.

Original: Receptive ﬁeld size of ﬁlters: As explained before, we can use more than one layer of ﬁlter while computing the frequency-based representation.
Revised: Receptive ﬁeld size of ﬁlters: As explained before, we can use more than one layer of ﬁlter while computing the frequency-based representation.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4 (left), one can have a second layer of ﬁlters which take as input the ﬁlter bank responses (g), and repeats the same process again.
Revised: 4 (left), one can have a second layer of ﬁlters which take as input the ﬁlter bank responses (g), and repeats the same process again.

Original: The ﬁnal frequency based representation will be the concatenation of representation after ﬁrst and second stage.
Revised: The ﬁnal frequency based representation will be the concatenation of representation after ﬁrst and second stage.

Original: In our experiment, we always use ﬁlters of size 3x3 (unless stated otherwise), stride = 1, padding = 0.
Revised: In our experiment, we always use ﬁlters of size 3x3 (unless stated otherwise), stride = 1, padding = 0.

Original: The number of ﬁlters in the ﬁrst, second and third layers are 64, 128 andrespectively.
Revised: The number of ﬁlters in the ﬁrst, second and third layers are 64, 128 andrespectively.

Original: We experiment with 4 settings - (i) 1 layer of 1x1 ﬁlters; (ii) 1 layer of 3x3 ﬁlters; (iii) 2 layers of 3x3 ﬁlters; and (iv) 3 layers of 3x3 ﬁlters.
Revised: We experiment with 4 settings - (i) 1 layer of 1x1 ﬁlters; (ii) 1 layer of 3x3 ﬁlters; (iii) 2 layers of 3x3 ﬁlters; and (iv) 3 layers of 3x3 ﬁlters.

Original: Between each layer is a combination of
Revised: Between each layer is a combination of

Original: ReLU non-linearity and maxpool operation (which further helps in increasing
Revised: ReLU non-linearity and maxpool operation (which further helps in increasing

Original: the effective receptive
Revised: the effective receptive

Original: [Figure]
Revised: [Figure]

Original: ﬁeld).
Revised: ﬁeld).

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: With setting (i), i.e. 1 layer of 1x1 ﬁlters, the only additional constrain introduced by L filter and L hybrid is to ensure consistency in 1x1 regions, i.e. RGB pixel distribution.
Revised: With setting (i), i.e. 1 layer of 1x1 ﬁlters, the only additional constrain introduced by L filter and L hybrid is to ensure consistency in 1x1 regions, i.e. RGB pixel distribution.

Original: In other words, this setting could be thought of as enforcing color histogram similarity between source and target images.
Revised: In other words, this setting could be thought of as enforcing color histogram similarity between source and target images.

Original: We observe a similar behavior in the results as well, where it fails to accurately transfer the tiger’s texture to other animals’ faces, but is able to faithfully transfer the color distribution.
Revised: We observe a similar behavior in the results as well, where it fails to accurately transfer the tiger’s texture to other animals’ faces, but is able to faithfully transfer the color distribution.

Original: Furthermore, the source dog in the second row has two characteristic property; brown patches above the eyes, and a white region in the middle going all the way to the forehead.
Revised: Furthermore, the source dog in the second row has two characteristic property; brown patches above the eyes, and a white region in the middle going all the way to the forehead.

Original: Transferring this particular appearance doesn’t always preserve the properties; e.g. (i) the tiger only has a white patch near the mouth, which doesn’t extend towards the forehead, (ii) the brown patches over the eyes are no longer transferred to the tiger’s face.
Revised: Transferring this particular appearance doesn’t always preserve the properties; e.g. (i) the tiger only has a white patch near the mouth, which doesn’t extend towards the forehead, (ii) the brown patches over the eyes are no longer transferred to the tiger’s face.

Original: In general, we observe that these properties (white patch, texture pattern), which extend beyond color distribution, are better captured with increasing receptive ﬁeld.
Revised: In general, we observe that these properties (white patch, texture pattern), which extend beyond color distribution, are better captured with increasing receptive ﬁeld.

Original: Setting (iii) and (iv) result in similar performance, so we choose the setting (iii) (a simpler version) as our ﬁnal model.
Revised: Setting (iii) and (iv) result in similar performance, so we choose the setting (iii) (a simpler version) as our ﬁnal model.

Original: Parameters optimized using L hybrid : We explain in sec.
Revised: Parameters optimized using L hybrid : We explain in sec.

Original: 3.3 that L hybrid is optimized by adjusting the generator weights.
Revised: 3.3 that L hybrid is optimized by adjusting the generator weights.

Original: However, the base model that we use (FineGAN) generates the foreground in two stages: the shape stage generating the outline, and the appearance stage ﬁlling in the details.
Revised: However, the base model that we use (FineGAN) generates the foreground in two stages: the shape stage generating the outline, and the appearance stage ﬁlling in the details.

Original: In animals ↔ animals, the shape stage often captures some appearance details as well (e.g. the leopard’s texture), and hence only optimizing the appearance stage doesn’t let the model transfer those characteristic details, since they are not being generated at the appearance stage (Fig.
Revised: In animals ↔ animals, the shape stage often captures some appearance details as well (e.g. the leopard’s texture), and hence only optimizing the appearance stage doesn’t let the model transfer those characteristic details, since they are not being generated at the appearance stage (Fig.

Original: Optimizing both the shape and appearance stage helps get rid of that issue, where the generation of appearance details gets properly shifted to the appearance stage, resulting in more accurate appearance transfer (Fig. 8 right).
Revised: Optimizing both the shape and appearance stage helps get rid of that issue, where the generation of appearance details gets properly shifted to the appearance stage, resulting in more accurate appearance transfer (Fig. 8 right).

Original: Effect of τ : NT - Xent (Normalized temperature-scaled cross entropy loss), which is used in both
Revised: Effect of τ : NT - Xent (Normalized temperature-scaled cross entropy loss), which is used in both

Original: L hybrid and L filter , uses τ as the temperature hypermarameter.
Revised: L hybrid and L filter , uses τ as the temperature hypermarameter.

Original: For all the experiments up until now,
Revised: For all the experiments up until now,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Shape
Revised: Shape

Original: Output  = 0.
Revised: Output  = 0.

Original:  = 0.  = 0.
Revised:  = 0.  = 0.

Original: we choose τ = 0 .
Revised: we choose τ = 0 .

Original: 5 as the default value.
Revised: 5 as the default value.

Original: In this section, we study whether our method’s performance is sensitive to the value used for τ .
Revised: In this section, we study whether our method’s performance is sensitive to the value used for τ .

Original: Results are presented in Fig.
Revised: Results are presented in Fig.

Original: 9, where we consider the birds ↔ cars setting, and run our algorithm for three values of τ , 0.1, 0.5 and 0.9.
Revised: 9, where we consider the birds ↔ cars setting, and run our algorithm for three values of τ , 0.1, 0.5 and 0.9.

Original: We observe that the ability to accurately transfer the appearance from one object to another remains largely consistent across the three cases.
Revised: We observe that the ability to accurately transfer the appearance from one object to another remains largely consistent across the three cases.

Original: This indicates that our method is ﬂexible, where one doesn’t need to tune the value of τ for accurate inter-domain disentanglement.
Revised: This indicates that our method is ﬂexible, where one doesn’t need to tune the value of τ for accurate inter-domain disentanglement.

Original: A. R ESISTIVITY ISSUE
Revised: A. R ESISTIVITY ISSUE

Original: In Fig.
Revised: In Fig.

Original: 10, we see that FineGAN shows a resistance towards changing the appearance of an animal; e.g. in row 4, the leopard in the target image only borrows a blackish color tone, and most of the signature leopard appearance remains as it is.
Revised: 10, we see that FineGAN shows a resistance towards changing the appearance of an animal; e.g. in row 4, the leopard in the target image only borrows a blackish color tone, and most of the signature leopard appearance remains as it is.

Original: In contrast, our method is able to comprehensively change the appearance of an animal; e.g. in the same row, leopard can shed its appearance and borrow everything from the tricolored dog, while also preserving any potential correspondence (the resulting animal retains the brown patch above the eye).
Revised: In contrast, our method is able to comprehensively change the appearance of an animal; e.g. in the same row, leopard can shed its appearance and borrow everything from the tricolored dog, while also preserving any potential correspondence (the resulting animal retains the brown patch above the eye).

Original: We study the resistivity property in more detail: for each latent shape vector x
Revised: We study the resistivity property in more detail: for each latent shape vector x

Original: i , we construct a list of images ( L ) by combining it with
Revised: i , we construct a list of images ( L ) by combining it with

Original: [Figure]
Revised: [Figure]

Original: all possible latent appearance vectors, keeping the latent noise ( z capturing pose) and background vectors the same, i.e. L = { G ( x i , y j , z i , b i ) } ∨ y j ∈ animals ↔ animals .
Revised: all possible latent appearance vectors, keeping the latent noise ( z capturing pose) and background vectors the same, i.e. L = { G ( x i , y j , z i , b i ) } ∨ y j ∈ animals ↔ animals .

Original: We then stack the images in L , compute standard deviation across channels separately, and then average to generate a heatmap image.
Revised: We then stack the images in L , compute standard deviation across channels separately, and then average to generate a heatmap image.

Original: We repeat this process for all the latent shape vectors, and plot the histogram of standard deviation values.
Revised: We repeat this process for all the latent shape vectors, and plot the histogram of standard deviation values.

Original: Fig.
Revised: Fig.

Original: 10 illustrates this process, and compares the heatmap images generated using our approach vs FineGAN.
Revised: 10 illustrates this process, and compares the heatmap images generated using our approach vs FineGAN.

Original: This helps us visualize the resistance property, as the heatmap and histogram through FineGAN indicates much lower standard deviation of pixel differences compared to our approach, implying more diversity of visual appearance attainable using our method.
Revised: This helps us visualize the resistance property, as the heatmap and histogram through FineGAN indicates much lower standard deviation of pixel differences compared to our approach, implying more diversity of visual appearance attainable using our method.

Original: A. S HAPE / APPEARANCE DISENTANGLEMENT A.4. C OLOR / TEXTURE HISTOGRAM CONSTRUCTION DETAILS
Revised: A. S HAPE / APPEARANCE DISENTANGLEMENT A.4. C OLOR / TEXTURE HISTOGRAM CONSTRUCTION DETAILS

Original: We ﬁrst deﬁne the process of obtaining the image and texture based cluster centers (codebook), and then detail the process of constructing and comparing the respective histogram representations of two images.
Revised: We ﬁrst deﬁne the process of obtaining the image and texture based cluster centers (codebook), and then detail the process of constructing and comparing the respective histogram representations of two images.

Original: We sample 50,000 pixel (RGB) values randomly from the images belonging to a multi-domain dataset (e.g. birds ↔ cars ), and perform k-means clustering with k = 50 to get the color centers.
Revised: We sample 50,000 pixel (RGB) values randomly from the images belonging to a multi-domain dataset (e.g. birds ↔ cars ), and perform k-means clustering with k = 50 to get the color centers.

Original: To compute the texture centers, the images in the dataset are ﬁrst convolved with a MR8 (Leung &
Revised: To compute the texture centers, the images in the dataset are ﬁrst convolved with a MR8 (Leung &

Original: Malik, 2001) ﬁlter bank, consisting of 38 ﬁlters and 8 responses capturing edge, bar and rotationally symmetric features.
Revised: Malik, 2001) ﬁlter bank, consisting of 38 ﬁlters and 8 responses capturing edge, bar and rotationally symmetric features.

Original: After this step, the process is similar to computing the color histograms: we sample 50,000 points from the images, each point being 8-dimensional, and cluster them intocenters representing the texton codebook.
Revised: After this step, the process is similar to computing the color histograms: we sample 50,000 points from the images, each point being 8-dimensional, and cluster them intocenters representing the texton codebook.

Original: We repeat this process of computing the color and texture clusters for each multi-domain setting separately.
Revised: We repeat this process of computing the color and texture clusters for each multi-domain setting separately.

Original: For each latent appearance vector y i , we generate a source image I s =
Revised: For each latent appearance vector y i , we generate a source image I s =

Original: G ( x i , y i , b i , z i ) and a target image I t =
Revised: G ( x i , y i , b i , z i ) and a target image I t =

Original: G ( x j , y i , b i , z i ) such that x i , y i belong to one domain and x j to another (e.g. x i /x
Revised: G ( x j , y i , b i , z i ) such that x i , y i belong to one domain and x j to another (e.g. x i /x

Original: j represents bird/car shape).
Revised: j represents bird/car shape).

Original: We then use a semantic segmentation network (DeepLabv3 (Chen et al., 2017))
Revised: We then use a semantic segmentation network (DeepLabv3 (Chen et al., 2017))

Original: pre-trained on MS COCO (Lin et al., 2014) to extract foreground pixels, and compute and compare color/texture based histograms between source and target images.
Revised: pre-trained on MS COCO (Lin et al., 2014) to extract foreground pixels, and compute and compare color/texture based histograms between source and target images.

Original: Note that I t is likely to represent some visual concepts which were never seen by the segmentation network pre-trained on real data (e.g. car with dog’s fur in Fig. 6), which could result in unreliable segmentations from the network.
Revised: Note that I t is likely to represent some visual concepts which were never seen by the segmentation network pre-trained on real data (e.g. car with dog’s fur in Fig. 6), which could result in unreliable segmentations from the network.

Original: We circumvent that by generating an intermediate image G ( x j , y j , b j , z i ) such that both the appearance/shape factors belong to the same target domain.
Revised: We circumvent that by generating an intermediate image G ( x j , y j , b j , z i ) such that both the appearance/shape factors belong to the same target domain.

Original: We then run the pre-trained segmentation network on this image, and use the predicted foreground locations for I t (which has the same shape factor, x j ).
Revised: We then run the pre-trained segmentation network on this image, and use the predicted foreground locations for I t (which has the same shape factor, x j ).

Original: A.4. S HAPE DISENTANGLEMENT
Revised: A.4. S HAPE DISENTANGLEMENT

Original: As mentioned in Sec.
Revised: As mentioned in Sec.

Original: 4.2.1, one can compute the standard deviation image across the stack, apply an appropriate threshold (we use values between 0.1-0.3) to get a binary mask mostly focusing on the foreground, as shown in Fig.
Revised: 4.2.1, one can compute the standard deviation image across the stack, apply an appropriate threshold (we use values between 0.1-0.3) to get a binary mask mostly focusing on the foreground, as shown in Fig.

Original: A.4. E VALUATING PART CORRESPONDENCES IN APPEARANCE TRANSFER
Revised: A.4. E VALUATING PART CORRESPONDENCES IN APPEARANCE TRANSFER

Original: In Sec. 2, we discussed that when the two domains under consideration have part correspondences (e.g. eyes/mouth of dogs and fox), transferring appearance from one to another happens in a way where properties around common object parts remain similar.
Revised: In Sec. 2, we discussed that when the two domains under consideration have part correspondences (e.g. eyes/mouth of dogs and fox), transferring appearance from one to another happens in a way where properties around common object parts remain similar.

Original: We illustrate examples of this in Fig.animals ↔ animals, where (i) the brown patches near the eye of the dog get transferred to the similar region in husky (second last row), (ii) the brown patch near the cheeks of the dog get transferred to the cheeks of the leopard.
Revised: We illustrate examples of this in Fig.animals ↔ animals, where (i) the brown patches near the eye of the dog get transferred to the similar region in husky (second last row), (ii) the brown patch near the cheeks of the dog get transferred to the cheeks of the leopard.

Original: In this section, we study this property empirically.
Revised: In this section, we study this property empirically.

Original: Given an input image, we ﬁrst extract six keypoints (nose, two eyes, forehead center, two ears) using an animal face keypoint detector.
Revised: Given an input image, we ﬁrst extract six keypoints (nose, two eyes, forehead center, two ears) using an animal face keypoint detector.

Original: For each such keypoint, we consider a small patch of 16 x 16 around it, and compute the color/texton histogram for this region.
Revised: For each such keypoint, we consider a small patch of 16 x 16 around it, and compute the color/texton histogram for this region.

Original: The process is similarly repeated for all the keypoints, resulting in six color histograms.
Revised: The process is similarly repeated for all the keypoints, resulting in six color histograms.

Original: When this appearance is transferred to a new image, the process is repeated again, by getting six histograms for the output.
Revised: When this appearance is transferred to a new image, the process is repeated again, by getting six histograms for the output.

Original: Finally, we compare the color histograms of corresponding keypoints in input/output using χ 2 -distance.
Revised: Finally, we compare the color histograms of corresponding keypoints in input/output using χ 2 -distance.

Original: If the part-level correspondences are preserved during the appearance transfer, then this distance should be low.
Revised: If the part-level correspondences are preserved during the appearance transfer, then this distance should be low.

Original: The overall process of computing and comparing the histograms is similar to one described in Sec.
Revised: The overall process of computing and comparing the histograms is similar to one described in Sec.

Original: A.4.1.
Revised: A.4.1.

Original: However, since the animals ↔ animals setting consists of a large diversity of species, there are cases where the pre-trained keypoint detector exhibits low conﬁdence while making predictions.
Revised: However, since the animals ↔ animals setting consists of a large diversity of species, there are cases where the pre-trained keypoint detector exhibits low conﬁdence while making predictions.

Original: We choose to not consider these cases during the evaluation, since it could negatively impact a method’s performance.
Revised: We choose to not consider these cases during the evaluation, since it could negatively impact a method’s performance.

Original: For color histogram similarity, FineGAN gets a score ( χ 2 -distance) of 0 .
Revised: For color histogram similarity, FineGAN gets a score ( χ 2 -distance) of 0 .

Original: 6197 ± 0 .
Revised: 6197 ± 0 .

Original: 26 , whereas our method gets 0 .
Revised: 26 , whereas our method gets 0 .

Original: 5404 ± 0 .
Revised: 5404 ± 0 .

Original: The trend holds for texton based histograms as well, where FineGAN gets a score of 0 .
Revised: The trend holds for texton based histograms as well, where FineGAN gets a score of 0 .

Original: 7851 ± 0 .
Revised: 7851 ± 0 .

Original: 22 , and our method gets 0 .
Revised: 22 , and our method gets 0 .

Original: 7178 ± 0 .
Revised: 7178 ± 0 .

Original: So, in general, our method results in a transfer where color/texture frequency around points of interest (e.g. eyes) remain more similar than FineGAN.
Revised: So, in general, our method results in a transfer where color/texture frequency around points of interest (e.g. eyes) remain more similar than FineGAN.

Original: A. U NDERSTANDING HYBRID IMAGES THROUGH IMAGE CLASSIFIERS
Revised: A. U NDERSTANDING HYBRID IMAGES THROUGH IMAGE CLASSIFIERS

Original: A high level goal of this work was to create hybrid, out-of-distribution images which do not exist in any domain exclusively.
Revised: A high level goal of this work was to create hybrid, out-of-distribution images which do not exist in any domain exclusively.

Original: We studied our method’s ability to do so in terms of its ability to accurately transfer properties (shape/appearance) across different domains.
Revised: We studied our method’s ability to do so in terms of its ability to accurately transfer properties (shape/appearance) across different domains.

Original: Orthogonal to this, however, could we use image classiﬁcation as a proxy to understand hybridness ?
Revised: Orthogonal to this, however, could we use image classiﬁcation as a proxy to understand hybridness ?

Original: Let us look at one of the results of FineGAN, from Fig.
Revised: Let us look at one of the results of FineGAN, from Fig.

Original: 6 row 4, dogs ↔ cars.
Revised: 6 row 4, dogs ↔ cars.

Original: The resulting car generated upon transferring the appearance of dog onto a car still looks like a realistic car, i.e. could potentially belong to the real dataset of cars.
Revised: The resulting car generated upon transferring the appearance of dog onto a car still looks like a realistic car, i.e. could potentially belong to the real dataset of cars.

Original: We posit that our method better transfers the semantics of the dog to a car; consequently, we expect the hybrid car and the dog to be more similar in some semantic feature space using our method, compared to FineGAN.
Revised: We posit that our method better transfers the semantics of the dog to a car; consequently, we expect the hybrid car and the dog to be more similar in some semantic feature space using our method, compared to FineGAN.

Original: We test this as follows - given an in-domain image ( x and y are tied) I =
Revised: We test this as follows - given an in-domain image ( x and y are tied) I =

Original: [Table]
Revised: [Table]

Original: G ( x, y, b, z ) (real brown dog), we combine its appearance ( y ) with factors from other domain to create a hybrid image, I
Revised: G ( x, y, b, z ) (real brown dog), we combine its appearance ( y ) with factors from other domain to create a hybrid image, I

Original: a = G ( x (cid:48) , y, b (cid:48) , z ) (the hybrid car).
Revised: a = G ( x (cid:48) , y, b (cid:48) , z ) (the hybrid car).

Original: We pass both these images through an image classiﬁer (we use VGG 16) pretrained on ImageNet Deng et al.
Revised: We pass both these images through an image classiﬁer (we use VGG 16) pretrained on ImageNet Deng et al.

Original: (2009), and compute cosine similarity between the feature representation of these two images.
Revised: (2009), and compute cosine similarity between the feature representation of these two images.

Original: A high similarity implies better transfer of semantics, i.e. a better hybrid image.
Revised: A high similarity implies better transfer of semantics, i.e. a better hybrid image.

Original: We repeat this experiment for all the indomain images, and present the averaged results in Table 3 (left); we see that our method consistently achieves higher similarity, especially for cars ↔ dogs and animals ↔ animals.
Revised: We repeat this experiment for all the indomain images, and present the averaged results in Table 3 (left); we see that our method consistently achieves higher similarity, especially for cars ↔ dogs and animals ↔ animals.

Original: Analogously, if transferring the dog’s appearance did make the car more hybrid, its similarity with the original car should decrease.
Revised: Analogously, if transferring the dog’s appearance did make the car more hybrid, its similarity with the original car should decrease.

Original: That is, given the same in-domain image G ( x, y, b, z ) (original car), we borrow appearance from the other domain y (cid:48) to create the hybrid image I s =
Revised: That is, given the same in-domain image G ( x, y, b, z ) (original car), we borrow appearance from the other domain y (cid:48) to create the hybrid image I s =

Original: G ( x, y (cid:48) , b, z ) .
Revised: G ( x, y (cid:48) , b, z ) .

Original: We repeat the previous experiment and get the cosine distance between their feature representation.
Revised: We repeat the previous experiment and get the cosine distance between their feature representation.

Original: Lower similarity implies that the hybrid image shifted away from the original image.
Revised: Lower similarity implies that the hybrid image shifted away from the original image.

Original: Averaged results (over all in-domain images) are presented in Table 3 (Right).
Revised: Averaged results (over all in-domain images) are presented in Table 3 (Right).

Original: We again observe that our method gets lower similarity score, implying more shift from a domain towards hybrid distribution.
Revised: We again observe that our method gets lower similarity score, implying more shift from a domain towards hybrid distribution.

Original: A. Q UALITATIVE RESULTS
Revised: A. Q UALITATIVE RESULTS

Original: Fig.
Revised: Fig.

Original: 12 demonstrates some more results from our method on different multi-domain settings discussed so far.
Revised: 12 demonstrates some more results from our method on different multi-domain settings discussed so far.

Original: A. C OMPARISONS TO S TAR GAN V
Revised: A. C OMPARISONS TO S TAR GAN V

Original: Fig.
Revised: Fig.

Original: 14 illustrates the results of StarGANv2 and our method.
Revised: 14 illustrates the results of StarGANv2 and our method.

Original: The key difference here is the properties that a method learns to disentangle: StarGANv2 can take the pose (the original paper uses the term “style” for this) of an animal (source image), and transfer it to a different animal (reference).
Revised: The key difference here is the properties that a method learns to disentangle: StarGANv2 can take the pose (the original paper uses the term “style” for this) of an animal (source image), and transfer it to a different animal (reference).

Original: So, in the resulting generated image, both animal’s shape and appearance come from one image (reference) and only pose comes from a different one.
Revised: So, in the resulting generated image, both animal’s shape and appearance come from one image (reference) and only pose comes from a different one.

Original: For example, the lion (last row) when borrowing properties from different animals still remains a lion while changing poses: it’s shape doesn’t become that of a cat (ﬁrst column).
Revised: For example, the lion (last row) when borrowing properties from different animals still remains a lion while changing poses: it’s shape doesn’t become that of a cat (ﬁrst column).

Original: Our method goes a step further, where it can also disentangle shape from appearance.
Revised: Our method goes a step further, where it can also disentangle shape from appearance.

Original: For example, when transferring the appearance of a tiger to a dog (second column), the resulting image precisely preserves the dog shape (e.g. its signature long ears, tongue), and gets the tiger’s appearance.
Revised: For example, when transferring the appearance of a tiger to a dog (second column), the resulting image precisely preserves the dog shape (e.g. its signature long ears, tongue), and gets the tiger’s appearance.

Original: So, our method has more control over different factors of an image.
Revised: So, our method has more control over different factors of an image.

Original: We further study StarGANv2’s disentanglement abilities, by using birds, cars and dogs as the three domains in the image-to-image translation setup.
Revised: We further study StarGANv2’s disentanglement abilities, by using birds, cars and dogs as the three domains in the image-to-image translation setup.

Original: The question we ask is the following: can the model learn to combine a source bird and a reference car, so that the resulting generation has the shape of the bird but appearance of the car?
Revised: The question we ask is the following: can the model learn to combine a source bird and a reference car, so that the resulting generation has the shape of the bird but appearance of the car?

Original: Fig.
Revised: Fig.

Original: 13 illustrates many such { source, reference } combinations which are used as input.
Revised: 13 illustrates many such { source, reference } combinations which are used as input.

Original: We observe that the reference image dictates most of the properties of the generated images, with the style image only (slightly) altering the object pose.
Revised: We observe that the reference image dictates most of the properties of the generated images, with the style image only (slightly) altering the object pose.

Original: The results in this case again demonstrate that StarGANv2 has issues capturing shape from one domain (e.g. style image) and appearance from another (e.g. reference image).
Revised: The results in this case again demonstrate that StarGANv2 has issues capturing shape from one domain (e.g. style image) and appearance from another (e.g. reference image).

Original: In contrast, we present our method speciﬁcally for this purpose, where the results shown in Fig.
Revised: In contrast, we present our method speciﬁcally for this purpose, where the results shown in Fig.

Original: 12 and 6 demonstrates its ability in learning inter-domain disentanglement of shape and appearance.
Revised: 12 and 6 demonstrates its ability in learning inter-domain disentanglement of shape and appearance.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]