Original: I  NSTANCE - AWARE I MAGE- TO -I MAGE T RANSLATION
Revised: I NSTA GAN: I NSTANCE - AWARE I MAGE T RANSLATION

Original: A BSTRACT
Revised: A BSTRACT

Original: Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs).
Revised: Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs).

Original: However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves signiﬁcant changes in shape , e.g., translating pants to skirts in fashion images.
Revised: However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves signiﬁcant changes in shape , e.g., translating pants to skirts in fashion images.

Original: To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN) , that incorporates the instance information ( e.g., object segmentation masks) and improves multi-instance transﬁguration .
Revised: To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN) , that incorporates the instance information ( e.g., object segmentation masks) and improves multi-instance transﬁguration .

Original: The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances.
Revised: The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances.

Original: To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances.
Revised: To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances.

Original: We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances.
Revised: We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances.

Original: Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases.
Revised: Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases.

Original: Code and results are available in https://github.com/sangwoomo/instagan .
Revised: Code and results are available in https://github.com/sangwoomo/instagan .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Cross-domain generation arises in many machine learning tasks, including neural machine translation (Artetxe et al., 2017; Lample et al., 2017), image synthesis (Reed et al., 2016; Zhu et al., 2016), text style transfer (Shen et al., 2017), and video generation (Bansal et al., 2018;
Revised: Cross-domain generation arises in many machine learning tasks, including neural machine translation (Artetxe et al., 2017; Lample et al., 2017), image synthesis (Reed et al., 2016; Zhu et al., 2016), text style transfer (Shen et al., 2017), and video generation (Bansal et al., 2018;

Original: Wang et al., 2018a; Chan et al., 2018).
Revised: Wang et al., 2018a; Chan et al., 2018).

Original: In particular, the unpaired (or unsupervised) image-to-image translation has achieved an impressive progress based on variants of generative adversarial networks (GANs) (Zhu et al., 2017; Liu et al., 2017; Choi et al., 2017; Almahairi et al., 2018;
Revised: In particular, the unpaired (or unsupervised) image-to-image translation has achieved an impressive progress based on variants of generative adversarial networks (GANs) (Zhu et al., 2017; Liu et al., 2017; Choi et al., 2017; Almahairi et al., 2018;

Original: Huang et al., 2018;
Revised: Huang et al., 2018;

Original: Lee et al., 2018), and has also drawn considerable attention due to its practical applications including colorization (Zhang et al., 2016), super-resolution (Ledig et al., 2017), semantic manipulation (Wang et al., 2018b), and domain adaptation (Bousmalis et al., 2017; Shrivastava et al., 2017; Hoffman et al., 2017).
Revised: Lee et al., 2018), and has also drawn considerable attention due to its practical applications including colorization (Zhang et al., 2016), super-resolution (Ledig et al., 2017), semantic manipulation (Wang et al., 2018b), and domain adaptation (Bousmalis et al., 2017; Shrivastava et al., 2017; Hoffman et al., 2017).

Original: Previous methods on this line of research, however, often fail on challenging tasks, in particular, when the translation task involves signiﬁcant changes in shape of instances (Zhu et al., 2017) or the images to translate contains multiple target instances (Gokaslan et al., 2018).
Revised: Previous methods on this line of research, however, often fail on challenging tasks, in particular, when the translation task involves signiﬁcant changes in shape of instances (Zhu et al., 2017) or the images to translate contains multiple target instances (Gokaslan et al., 2018).

Original: Our goal is to extend image-to-image translation towards such challenging tasks, which can strengthen its applicability up to the next level, e.g., changing pants to skirts in fashion images for a customer to decide which one is better to buy.
Revised: Our goal is to extend image-to-image translation towards such challenging tasks, which can strengthen its applicability up to the next level, e.g., changing pants to skirts in fashion images for a customer to decide which one is better to buy.

Original: To this end, we propose a novel method that incorporates the instance information of multiple target objectsin the framework of generative adversarial networks (GAN); hence we called it instance-aware GAN (InstaGAN) .
Revised: To this end, we propose a novel method that incorporates the instance information of multiple target objectsin the framework of generative adversarial networks (GAN); hence we called it instance-aware GAN (InstaGAN) .

Original: In this work, we use the object segmentation masks for instance information, which may be a good representation for instance shapes, as it contains object boundaries while ignoring other details such as color.
Revised: In this work, we use the object segmentation masks for instance information, which may be a good representation for instance shapes, as it contains object boundaries while ignoring other details such as color.

Original: Using the information, our method shows impressive results for multi-instance transﬁguration tasks, as shown in Figure 1.
Revised: Using the information, our method shows impressive results for multi-instance transﬁguration tasks, as shown in Figure 1.

Original: Our main contribution is three-fold: an instance-augmented neural architecture, a context preserving loss, and a sequential mini-batch inference/training technique.
Revised: Our main contribution is three-fold: an instance-augmented neural architecture, a context preserving loss, and a sequential mini-batch inference/training technique.

Original: First, we propose a neural network architecture that translates both an image and the corresponding set of instance attributes.
Revised: First, we propose a neural network architecture that translates both an image and the corresponding set of instance attributes.

Original: Our architecture can translate an arbitrary number of instance attributes conditioned by the input, and is designed to be permutation-invariant to the order of instances.
Revised: Our architecture can translate an arbitrary number of instance attributes conditioned by the input, and is designed to be permutation-invariant to the order of instances.

Original: Second, we propose a context preserving loss that encourages the network to focus on target instances in translation and learn an identity function outside of them.
Revised: Second, we propose a context preserving loss that encourages the network to focus on target instances in translation and learn an identity function outside of them.

Original: Namely, it aims at preserving the background context while transforming the target instances.
Revised: Namely, it aims at preserving the background context while transforming the target instances.

Original: Finally, we propose a sequential mini-batch inference/training technique, i.e., translating the mini-batches of instance attributes sequentially, instead of doing the entire set at once.
Revised: Finally, we propose a sequential mini-batch inference/training technique, i.e., translating the mini-batches of instance attributes sequentially, instead of doing the entire set at once.

Original: It allows to handle a large number of instance attributes with a limited GPU memory, and thus enhances the network to generalize better for images with many instances.
Revised: It allows to handle a large number of instance attributes with a limited GPU memory, and thus enhances the network to generalize better for images with many instances.

Original: Furthermore, it improves the translation quality of images with even a few instances because it acts as data augmentation during training by producing multiple intermediate samples.
Revised: Furthermore, it improves the translation quality of images with even a few instances because it acts as data augmentation during training by producing multiple intermediate samples.

Original: All the aforementioned contributions are dedicated to how to incorporates the instance information ( e.g., segmentation masks) for image-to-image translation.
Revised: All the aforementioned contributions are dedicated to how to incorporates the instance information ( e.g., segmentation masks) for image-to-image translation.

Original: However, we believe that our approach is applicable to numerous other cross-domain generation tasks where set-structured side information is available.
Revised: However, we believe that our approach is applicable to numerous other cross-domain generation tasks where set-structured side information is available.

Original: To the best of our knowledge, we are the ﬁrst to report image-to-image translation results for multiinstance transﬁguration tasks.
Revised: To the best of our knowledge, we are the ﬁrst to report image-to-image translation results for multiinstance transﬁguration tasks.

Original: A few number of recent methods (Kim et al., 2017; Liu et al., 2017;
Revised: A few number of recent methods (Kim et al., 2017; Liu et al., 2017;

Original: Gokaslan et al., 2018) show some transﬁguration results but only for images with a single instance often in a clear background.
Revised: Gokaslan et al., 2018) show some transﬁguration results but only for images with a single instance often in a clear background.

Original: Unlike the previous results in a simple setting, our focus is on the harmony of instances naturally rendered with the background.
Revised: Unlike the previous results in a simple setting, our focus is on the harmony of instances naturally rendered with the background.

Original: On the other hand, CycleGAN (Zhu et al., 2017) show some results for multi-instance cases, but report only a limited performance for transﬁguration tasks.
Revised: On the other hand, CycleGAN (Zhu et al., 2017) show some results for multi-instance cases, but report only a limited performance for transﬁguration tasks.

Original: At a high level, the signiﬁcance of our work is also on discovering that the instance information is effective for shape-transforming image-to-image translation, which we think would be inﬂuential to other related research in the future.
Revised: At a high level, the signiﬁcance of our work is also on discovering that the instance information is effective for shape-transforming image-to-image translation, which we think would be inﬂuential to other related research in the future.

Original: Mask contrast-GAN (Liang et al., 2017) and Attention-GAN (Mejjati et al., 2018) use segmentation masks or predicted attentions, but only to attach the background to the (translated) cropped instances.
Revised: Mask contrast-GAN (Liang et al., 2017) and Attention-GAN (Mejjati et al., 2018) use segmentation masks or predicted attentions, but only to attach the background to the (translated) cropped instances.

Original: They do not allow to transform the shapes of the instances.
Revised: They do not allow to transform the shapes of the instances.

Original: To the contrary, our method learns how to preserve the background by optimizing the context preserving loss, thus facilitating the shape transformation.
Revised: To the contrary, our method learns how to preserve the background by optimizing the context preserving loss, thus facilitating the shape transformation.

Original:  I NSTA GAN: I NSTANCE - AWARE I MAGE - TO -I MAGE T RANSLATION
Revised:  I NSTA GAN: I NSTANCE - AWARE I MAGE - TO -I MAGE T RANSLATION

Original: Given two image domains X and Y , the problem of image-to-image translation aims to learn mappings across different image domains, G XY :
Revised: Given two image domains X and Y , the problem of image-to-image translation aims to learn mappings across different image domains, G XY :

Original: X → Y or/and G YX : Y → X , i.e., transforming target scene elements while preserving the original contexts.
Revised: X → Y or/and G YX : Y → X , i.e., transforming target scene elements while preserving the original contexts.

Original: This can also be formulated as a conditional generative modeling task where we estimate the conditionals p ( y | x ) or/and p ( x | y ) .
Revised: This can also be formulated as a conditional generative modeling task where we estimate the conditionals p ( y | x ) or/and p ( x | y ) .

Original: The goal of unsupervised translation we tackle is to recover such mappings only using unpaired samples from marginal distributions of original data, p data ( x ) and p data ( y ) of two image domains.
Revised: The goal of unsupervised translation we tackle is to recover such mappings only using unpaired samples from marginal distributions of original data, p data ( x ) and p data ( y ) of two image domains.

Original: The main and unique idea of our approach is to incorporate the additional instance information, i.e., augment a space of set of instance attributes A to the original image space X , to improve the image-to-image translation.
Revised: The main and unique idea of our approach is to incorporate the additional instance information, i.e., augment a space of set of instance attributes A to the original image space X , to improve the image-to-image translation.

Original: The set of instance attributes a ∈ A comprises all individual attributes of N target instances: a = { a i } Ni =1 .
Revised: The set of instance attributes a ∈ A comprises all individual attributes of N target instances: a = { a i } Ni =1 .

Original: In this work, we use an instance segmentation mask only, but we remark that any useful type of instance information can be incorporated for the attributes.
Revised: In this work, we use an instance segmentation mask only, but we remark that any useful type of instance information can be incorporated for the attributes.

Original: Our approach then can be described as learning joint-mappings between attribute-augmented spaces
Revised: Our approach then can be described as learning joint-mappings between attribute-augmented spaces

Original: X ×A and Y ×B .
Revised: X ×A and Y ×B .

Original: This leads to disentangle different instances in the image and allows the generator to perform an accurate and detailed translation.
Revised: This leads to disentangle different instances in the image and allows the generator to perform an accurate and detailed translation.

Original: We learn our attribute-augmented mapping in the framework of generative adversarial networks (GANs) (Goodfellow et al., 2014), hence, we call it instance-aware GAN (InstaGAN) .
Revised: We learn our attribute-augmented mapping in the framework of generative adversarial networks (GANs) (Goodfellow et al., 2014), hence, we call it instance-aware GAN (InstaGAN) .

Original: We present details of our approach in the following subsections.
Revised: We present details of our approach in the following subsections.

Original: 2. I NSTA GAN A RCHITECTURE
Revised: 2. I NSTA GAN A RCHITECTURE

Original: Recent GAN-based methods (Zhu et al., 2017; Liu et al., 2017) have achieved impressive performance in the unsupervised translation by jointly training two coupled mappings G XY and G YX with
Revised: Recent GAN-based methods (Zhu et al., 2017; Liu et al., 2017) have achieved impressive performance in the unsupervised translation by jointly training two coupled mappings G XY and G YX with

Original: [Figure]
Revised: [Figure]

Original: a cycle-consistency loss that encourages G YX ( G XY ( x ))
Revised: a cycle-consistency loss that encourages G YX ( G XY ( x ))

Original: ≈ x and G XY ( G YX ( y ))
Revised: ≈ x and G XY ( G YX ( y ))

Original: ≈ y .
Revised: ≈ y .

Original: Namely, we choose to leverage the CycleGAN approach (Zhu et al., 2017) to build our InstaGAN.
Revised: Namely, we choose to leverage the CycleGAN approach (Zhu et al., 2017) to build our InstaGAN.

Original: However, we remark that training two coupled mappings is not essential for our method, and one can also design a single mapping following other approaches (Benaim & Wolf, 2017; Galanti et al., 2018).
Revised: However, we remark that training two coupled mappings is not essential for our method, and one can also design a single mapping following other approaches (Benaim & Wolf, 2017; Galanti et al., 2018).

Original: Figure 2 illustrates the overall architecture of our model.
Revised: Figure 2 illustrates the overall architecture of our model.

Original: We train two coupled generators
Revised: We train two coupled generators

Original: G XY :
Revised: G XY :

Original: X ×
Revised: X ×

Original: A → Y × B and G YX : Y × B → X × A , where G XY translates the original data ( x, a ) to the target domain data ( y (cid:48) , b (cid:48) ) (and vice versa for G YX ), with adversarial discriminators
Revised: A → Y × B and G YX : Y × B → X × A , where G XY translates the original data ( x, a ) to the target domain data ( y (cid:48) , b (cid:48) ) (and vice versa for G YX ), with adversarial discriminators

Original: D X :
Revised: D X :

Original: X ×
Revised: X ×

Original: A → { ‘X’, ‘not X’ } and D Y : Y × B → { ‘Y’, ‘not Y’ } , where D X determines if the data (original ( x, a ) or translated ( x (cid:48) , a (cid:48) ) ) is in the target domain X × A
Revised: A → { ‘X’, ‘not X’ } and D Y : Y × B → { ‘Y’, ‘not Y’ } , where D X determines if the data (original ( x, a ) or translated ( x (cid:48) , a (cid:48) ) ) is in the target domain X × A

Original: or not (and vice versa for D Y ).
Revised: or not (and vice versa for D Y ).

Original: Our generator G encodes both x and a , and translates them into y (cid:48) and b (cid:48) .
Revised: Our generator G encodes both x and a , and translates them into y (cid:48) and b (cid:48) .

Original: Notably, the order of the instance attributes in the set a should not affect the translated image y (cid:48) , and each instance attribute in the set a should be translated to the corresponding one in b (cid:48) .
Revised: Notably, the order of the instance attributes in the set a should not affect the translated image y (cid:48) , and each instance attribute in the set a should be translated to the corresponding one in b (cid:48) .

Original: In other words, y (cid:48) is permutation-invariant with respect to the instances in a , and b (cid:48) is permutation-equivariant with respect to them.
Revised: In other words, y (cid:48) is permutation-invariant with respect to the instances in a , and b (cid:48) is permutation-equivariant with respect to them.

Original: These properties can be implemented by introducing proper operators in feature encoding (Zaheer et al., 2017).
Revised: These properties can be implemented by introducing proper operators in feature encoding (Zaheer et al., 2017).

Original: We ﬁrst extract individual features from image and attributes using image feature extractor f GX and attribute feature extractor f GA , respectively.
Revised: We ﬁrst extract individual features from image and attributes using image feature extractor f GX and attribute feature extractor f GA , respectively.

Original: The attribute features individually extracted using f GA are then aggregated into a permutation-invariant set feature via summation:
Revised: The attribute features individually extracted using f GA are then aggregated into a permutation-invariant set feature via summation:

Original: (cid:80) Ni =1 f GA ( a i ) .
Revised: (cid:80) Ni =1 f GA ( a i ) .

Original: As illustrated in Figure 2b, we concatenate some of image and attribute features with the set feature, and feed them to image and attribute generators.
Revised: As illustrated in Figure 2b, we concatenate some of image and attribute features with the set feature, and feed them to image and attribute generators.

Original: Formally, the image representation h GX and the n -th attribute representation h n GA in generator G can be formulated as:
Revised: Formally, the image representation h GX and the n -th attribute representation h n GA in generator G can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where each attribute encoding h n GA process features of all attributes as a contextual feature
Revised: where each attribute encoding h n GA process features of all attributes as a contextual feature

Original: Finally, h GX is fed to the image generator g GX , and h n GA ( n = 1 , . . .
Revised: Finally, h GX is fed to the image generator g GX , and h n GA ( n = 1 , . . .

Original: , N ) are to the attribute generator g GA .
Revised: , N ) are to the attribute generator g GA .

Original: On the other hand, our discriminator D encodes both x and a (or x (cid:48) and a (cid:48) ), and determines whether the pair is from the domain or not.
Revised: On the other hand, our discriminator D encodes both x and a (or x (cid:48) and a (cid:48) ), and determines whether the pair is from the domain or not.

Original: Here, the order of the instance attributes in the set a should not affect the output.
Revised: Here, the order of the instance attributes in the set a should not affect the output.

Original: In a similar manner above, our representation in discriminator D , which is permutation-invariant to the instances, is formulated as:
Revised: In a similar manner above, our representation in discriminator D , which is permutation-invariant to the instances, is formulated as:

Original: [Equation]
Revised: [Equation]

Original: which is fed to an adversarial discriminator g DX .
Revised: which is fed to an adversarial discriminator g DX .

Original: We emphasize that the joint encoding of both image x and instance attributes a for each neural component is crucial because it allows the network to learn the relation between x and a .
Revised: We emphasize that the joint encoding of both image x and instance attributes a for each neural component is crucial because it allows the network to learn the relation between x and a .

Original: For example, if two separate encodings and discriminators are used for x and a , the generator may be misled to produce image and instance masks that do not match with each other.
Revised: For example, if two separate encodings and discriminators are used for x and a , the generator may be misled to produce image and instance masks that do not match with each other.

Original: By using the joint encoding and discriminator, our generator can produce an image of instances properly depicted on the area consistent with its segmentation masks.
Revised: By using the joint encoding and discriminator, our generator can produce an image of instances properly depicted on the area consistent with its segmentation masks.

Original: As will be seen in Section 3, our approach can disentangle output instances considering their original layouts.
Revised: As will be seen in Section 3, our approach can disentangle output instances considering their original layouts.

Original: Note that any types of neural networks may be used for sub-network architectures mentioned above such as f GX , f GA , f DX , f DA , g GX , g GA , and g DX .
Revised: Note that any types of neural networks may be used for sub-network architectures mentioned above such as f GX , f GA , f DX , f DA , g GX , g GA , and g DX .

Original: We describe the detailed architectures used in our experiments in Appendix A.
Revised: We describe the detailed architectures used in our experiments in Appendix A.

Original: 2. T RAINING L OSS
Revised: 2. T RAINING L OSS

Original: Remind that an image-to-image translation model aims to translate a domain while keeping the original contexts ( e.g., background or instances’ domain-independent characteristics such as the looking direction).
Revised: Remind that an image-to-image translation model aims to translate a domain while keeping the original contexts ( e.g., background or instances’ domain-independent characteristics such as the looking direction).

Original: To this end, we both consider the domain loss, which makes the generated outputs to follow the style of a target domain, and the content loss, which makes the outputs to keep the original contents.
Revised: To this end, we both consider the domain loss, which makes the generated outputs to follow the style of a target domain, and the content loss, which makes the outputs to keep the original contents.

Original: Following our baseline model, CycleGAN (Zhu et al., 2017), we use the GAN loss for the domain loss, and consider both the cycle-consistency loss (Kim et al., 2017;
Revised: Following our baseline model, CycleGAN (Zhu et al., 2017), we use the GAN loss for the domain loss, and consider both the cycle-consistency loss (Kim et al., 2017;

Original: Yi et al., 2017) and the identity mapping loss (Taigman et al., 2016) for the content losses.
Revised: Yi et al., 2017) and the identity mapping loss (Taigman et al., 2016) for the content losses.

Original: In addition, we also propose a new content loss, coined context preserving loss , using the original and predicted segmentation information.
Revised: In addition, we also propose a new content loss, coined context preserving loss , using the original and predicted segmentation information.

Original: In what follows, we formally deﬁne our training loss in detail.
Revised: In what follows, we formally deﬁne our training loss in detail.

Original: For simplicity, we denote our loss function as a function of a single training sample ( x, a ) ∈
Revised: For simplicity, we denote our loss function as a function of a single training sample ( x, a ) ∈

Original: X × A
Revised: X × A

Original: and ( y, b ) ∈ Y × B , while one has to minimize its empirical means in training.
Revised: and ( y, b ) ∈ Y × B , while one has to minimize its empirical means in training.

Original: The GAN loss is originally proposed by Goodfellow et al.
Revised: The GAN loss is originally proposed by Goodfellow et al.

Original: (2014) for generative modeling via alternately training generator G and discriminator D .
Revised: (2014) for generative modeling via alternately training generator G and discriminator D .

Original: Here, D determines if the data is a real one of a fake/generated/translated one made by G .
Revised: Here, D determines if the data is a real one of a fake/generated/translated one made by G .

Original: There are numerous variants of the GAN loss (Nowozin et al., 2016; Arjovsky et al., 2017; Li et al., 2017;
Revised: There are numerous variants of the GAN loss (Nowozin et al., 2016; Arjovsky et al., 2017; Li et al., 2017;

Original: Mroueh et al., 2017), and we follow the LSGAN scheme (Mao et al., 2017), which is empirically known to show a stably good performance:
Revised: Mroueh et al., 2017), and we follow the LSGAN scheme (Mao et al., 2017), which is empirically known to show a stably good performance:

Original: [Equation]
Revised: [Equation]

Original: For keeping the original content, the cycle-consistency loss L cyc and the identity mapping loss L idt enforce samples not to lose the original information after translating twice and once, respectively:
Revised: For keeping the original content, the cycle-consistency loss L cyc and the identity mapping loss L idt enforce samples not to lose the original information after translating twice and once, respectively:

Original: [Equation]
Revised: [Equation]

Original: Finally, our newly proposed context preserving loss L ctx enforces to translate instances only, while keeping outside of them, i.e., background.
Revised: Finally, our newly proposed context preserving loss L ctx enforces to translate instances only, while keeping outside of them, i.e., background.

Original: Formally, it is a pixel-wise weighted (cid:96) 1 -loss where the weight is 1 for background and 0 for instances.
Revised: Formally, it is a pixel-wise weighted (cid:96) 1 -loss where the weight is 1 for background and 0 for instances.

Original: Here, note that backgrounds for two domains become different in transﬁguration-type translation involving signiﬁcant shape changes.
Revised: Here, note that backgrounds for two domains become different in transﬁguration-type translation involving signiﬁcant shape changes.

Original: Hence, we consider the non-zero weight only if a pixel is in background in both original and translated ones.
Revised: Hence, we consider the non-zero weight only if a pixel is in background in both original and translated ones.

Original: Namely, for the original samples ( x, a ) , ( y, b ) and the translated one ( y (cid:48) , b (cid:48) ) , ( x (cid:48) , a (cid:48) ) , we let the weight w ( a , b (cid:48) ) , w ( b , a (cid:48) ) be one minus the element-wise minimum of binary represented instance masks, and we propose
Revised: Namely, for the original samples ( x, a ) , ( y, b ) and the translated one ( y (cid:48) , b (cid:48) ) , ( x (cid:48) , a (cid:48) ) , we let the weight w ( a , b (cid:48) ) , w ( b , a (cid:48) ) be one minus the element-wise minimum of binary represented instance masks, and we propose

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) is the element-wise product.
Revised: where (cid:12) is the element-wise product.

Original: In our experiments, we found that the context preserving loss not only keeps the background better, but also improves the quality of generated instance segmentations.
Revised: In our experiments, we found that the context preserving loss not only keeps the background better, but also improves the quality of generated instance segmentations.

Original: Finally, the total loss of InstaGAN is
Revised: Finally, the total loss of InstaGAN is

Original: [Equation]
Revised: [Equation]

Original: where λ cyc , λ idt , λ ctx > 0 are some hyper-parameters balancing the losses.
Revised: where λ cyc , λ idt , λ ctx > 0 are some hyper-parameters balancing the losses.

Original: 2. S EQUENTIAL M INI -B ATCH T RANSLATION
Revised: 2. S EQUENTIAL M INI -B ATCH T RANSLATION

Original: While the proposed architecture is able to translate an arbitrary number of instances in principle, the GPU memory required linearly increases with the number of instances.
Revised: While the proposed architecture is able to translate an arbitrary number of instances in principle, the GPU memory required linearly increases with the number of instances.

Original: For example, in our experiments, a machine was able to forward only a small number (say, 2) of instance attributes during training, and thus the learned model suffered from poor generalization to images with a larger number of instances.
Revised: For example, in our experiments, a machine was able to forward only a small number (say, 2) of instance attributes during training, and thus the learned model suffered from poor generalization to images with a larger number of instances.

Original: To address this issue, we propose a new inference/training technique, which allows to train an arbitrary number of instances without increasing the GPU memory.
Revised: To address this issue, we propose a new inference/training technique, which allows to train an arbitrary number of instances without increasing the GPU memory.

Original: We ﬁrst describe the sequential inference scheme that translates the subset of instances sequentially, and then describe the corresponding mini-batch training technique.
Revised: We ﬁrst describe the sequential inference scheme that translates the subset of instances sequentially, and then describe the corresponding mini-batch training technique.

Original: Given an input ( x, a ) , we ﬁrst divide the set of instance masks a into mini-batches a 1 , . . .
Revised: Given an input ( x, a ) , we ﬁrst divide the set of instance masks a into mini-batches a 1 , . . .

Original: , a M , i.e., a = (cid:83) i a i and a i ∩ a j = ∅ for i (cid:54) =
Revised: , a M , i.e., a = (cid:83) i a i and a i ∩ a j = ∅ for i (cid:54) =

Original: j .
Revised: j .

Original: Then, at the m -th iteration for m = 1 , 2 , . . .
Revised: Then, at the m -th iteration for m = 1 , 2 , . . .

Original: , M , we translate the image-mask pair ( x m , a m ) , where x m is the translated image y (cid:48) m
Revised: , M , we translate the image-mask pair ( x m , a m ) , where x m is the translated image y (cid:48) m

Original: − 1 from the previous iteration, and x 1 = x .
Revised: − 1 from the previous iteration, and x 1 = x .

Original: In this sequential scheme, at each iteration, the generator G outputs an intermediate translated image y (cid:48) m , which accumulates all mini-batch translations up to the current iteration, and a translated mini-batch of instance masks b (cid:48) m :
Revised: In this sequential scheme, at each iteration, the generator G outputs an intermediate translated image y (cid:48) m , which accumulates all mini-batch translations up to the current iteration, and a translated mini-batch of instance masks b (cid:48) m :

Original: [Equation]
Revised: [Equation]

Original: In order to align the translated image with mini-batches of instance masks, we aggregate all the translated mini-batch and produce a translated sample:
Revised: In order to align the translated image with mini-batches of instance masks, we aggregate all the translated mini-batch and produce a translated sample:

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal output of the proposed sequential inference scheme is ( y (cid:48)
Revised: The ﬁnal output of the proposed sequential inference scheme is ( y (cid:48)

Original: M , b (cid:48) 1: M ) .
Revised: M , b (cid:48) 1: M ) .

Original: We also propose the corresponding sequential training algorithm, as illustrated in Figure 3.
Revised: We also propose the corresponding sequential training algorithm, as illustrated in Figure 3.

Original: We apply content loss (4-6) to the intermediate samples ( y (cid:48) m , b (cid:48) m ) of current mini-batch a m , as it is just a function of inputs and outputs of the generator G . 

Revised: We apply content loss (4-6) to the intermediate samples ( y (cid:48) m , b (cid:48) m ) of current mini-batch a m , as it is just a function of inputs and outputs of the generator G .

Original: In contrast, we apply GAN loss (3) to the samples of aggregated mini-batches ( y (cid:48) m , b (cid:48) 1: m ) , because the network fails to align images and masks when using only a partial subset of instance masks.
Revised: In contrast, we apply GAN loss (3) to the samples of aggregated mini-batches ( y (cid:48) m , b (cid:48) 1: m ) , because the network fails to align images and masks when using only a partial subset of instance masks.

Original: We used real/original samples { x } with the full set of instance masks only.
Revised: We used real/original samples { x } with the full set of instance masks only.

Original: Formally, the sequential version of the training loss of InstaGAN is
Revised: Formally, the sequential version of the training loss of InstaGAN is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where L content = λ cyc L cyc + λ idt L idt + λ ctx L ctx .
Revised: where L content = λ cyc L cyc + λ idt L idt + λ ctx L ctx .

Original: We detach every m -th iteration of training, i.e., backpropagating with the mini-batch a m , so that only a ﬁxed GPU memory is required, regardless of the number of training instances.
Revised: We detach every m -th iteration of training, i.e., backpropagating with the mini-batch a m , so that only a ﬁxed GPU memory is required, regardless of the number of training instances.

Original: 3 Hence, the
Revised: 3 Hence, the

Original: We still recommend users to increase the subset size as long as the sequential training allows for training with samples containing many instances, and thus improves the generalization performance.
Revised: We still recommend users to increase the subset size as long as the sequential training allows for training with samples containing many instances, and thus improves the generalization performance.

Original: Furthermore, it also improves translation of an image even with a few instances, compared to the one-step approach, due to its data augmentation effect using intermediate samples ( x m , a m ) .
Revised: Furthermore, it also improves translation of an image even with a few instances, compared to the one-step approach, due to its data augmentation effect using intermediate samples ( x m , a m ) .

Original: In our experiments, we divided the instances into mini-batches a 1 , . . .
Revised: In our experiments, we divided the instances into mini-batches a 1 , . . .

Original: , a M according to the decreasing order of the spatial sizes of instances.
Revised: , a M according to the decreasing order of the spatial sizes of instances.

Original: Interestingly, the decreasing order showed a better performance than the random order.
Revised: Interestingly, the decreasing order showed a better performance than the random order.

Original: We believe that this is because small instances tend to be occluded by other instances in images, thus often losing their intrinsic shape information.
Revised: We believe that this is because small instances tend to be occluded by other instances in images, thus often losing their intrinsic shape information.

Original:  E XPERIMENTAL R ESULTS 3. I MAGE - TO -I MAGE T RANSLATION R ESULTS
Revised:  E XPERIMENTAL R ESULTS 3. I MAGE - TO -I MAGE T RANSLATION R ESULTS

Original: We ﬁrst qualitatively evaluate our method on various datasets.
Revised: We ﬁrst qualitatively evaluate our method on various datasets.

Original: We compare our model, InstaGAN, with the baseline model, CycleGAN (Zhu et al., 2017).
Revised: We compare our model, InstaGAN, with the baseline model, CycleGAN (Zhu et al., 2017).

Original: For fair comparisons, we doubled the number of parameters of CycleGAN, as InstaGAN uses two networks for image and masks, respectively.
Revised: For fair comparisons, we doubled the number of parameters of CycleGAN, as InstaGAN uses two networks for image and masks, respectively.

Original: We sample two classes from various datasets, including clothing co-parsing (CCP)
Revised: We sample two classes from various datasets, including clothing co-parsing (CCP)

Original: (Yang et al.,
Revised: (Yang et al.,

Original: , multi-human parsing (MHP) (Zhao et al., 2018), and MS COCO (Lin et al., 2014) datasets, and use them as the two domains for translation.
Revised: , multi-human parsing (MHP) (Zhao et al., 2018), and MS COCO (Lin et al., 2014) datasets, and use them as the two domains for translation.

Original: In visualizations, we merge all instance masks into one for the sake of compactness.
Revised: In visualizations, we merge all instance masks into one for the sake of compactness.

Original: See Appendix B for detailed settings for our experiments.
Revised: See Appendix B for detailed settings for our experiments.

Original: The translation results for three datasets are presented in Figure 4, 5, and 6, respectively.
Revised: The translation results for three datasets are presented in Figure 4, 5, and 6, respectively.

Original: While
Revised: While

Original: CycleGAN mostly fails, our method generates reasonable shapes of the target instances and keeps the original contexts by focusing on the instances via the context preserving loss.
Revised: CycleGAN mostly fails, our method generates reasonable shapes of the target instances and keeps the original contexts by focusing on the instances via the context preserving loss.

Original: For example, see the results on sheep ↔ giraffe in Figure 6.
Revised: For example, see the results on sheep ↔ giraffe in Figure 6.

Original: CycleGAN often generates sheep-like instances but loses the original background.
Revised: CycleGAN often generates sheep-like instances but loses the original background.

Original: InstaGAN not only generates better sheep or giraffes, but also preserves the layout of the original instances, i.e., the looking direction (left, right, front) of sheep and giraffes are consistent after translation.
Revised: InstaGAN not only generates better sheep or giraffes, but also preserves the layout of the original instances, i.e., the looking direction (left, right, front) of sheep and giraffes are consistent after translation.

Original: More experimental results are presented in Appendix E. Code and results are available in https://github.com/sangwoomo/instagan .
Revised: More experimental results are presented in Appendix E. Code and results are available in https://github.com/sangwoomo/instagan .

Original: On the other hand, our method can control the instances to translate by conditioning the input, as shown in Figure 7.
Revised: On the other hand, our method can control the instances to translate by conditioning the input, as shown in Figure 7.

Original: Such a control is impossible under CycleGAN.
Revised: Such a control is impossible under CycleGAN.

Original: We also note that we focus on complex (multi-instance transﬁguration) tasks to emphasize the advantages of our method.
Revised: We also note that we focus on complex (multi-instance transﬁguration) tasks to emphasize the advantages of our method.

Original: Nevertheless, our method is also attractive to use even for simple tasks ( e.g., horse ↔ zebra) as it reduces false positives/negatives via the context preserving loss and enables to control translation.
Revised: Nevertheless, our method is also attractive to use even for simple tasks ( e.g., horse ↔ zebra) as it reduces false positives/negatives via the context preserving loss and enables to control translation.

Original: We ﬁnally emphasize that our method showed good results even when we use predicted segmentation for inference, as shown in Figure 8, and this can reduce the cost of collecting mask labels in practice.
Revised: We ﬁnally emphasize that our method showed good results even when we use predicted segmentation for inference, as shown in Figure 8, and this can reduce the cost of collecting mask labels in practice.

Original: Finally, we also quantitatively evaluate the translation performance of our method.
Revised: Finally, we also quantitatively evaluate the translation performance of our method.

Original: We measure the classiﬁcation score, the ratio of images predicted as the target class by a pretrained classiﬁer.
Revised: We measure the classiﬁcation score, the ratio of images predicted as the target class by a pretrained classiﬁer.

Original: Speciﬁcally, we ﬁne-tune the ﬁnal layers of the ImageNet (Deng et al., 2009) pretrained VGG- (Simonyan & Zisserman, 2014) network, as a binary classiﬁer for each domain.
Revised: Speciﬁcally, we ﬁne-tune the ﬁnal layers of the ImageNet (Deng et al., 2009) pretrained VGG- (Simonyan & Zisserman, 2014) network, as a binary classiﬁer for each domain.

Original: Table 1 and Tablein Appendix D show the classiﬁcation scores for CCP and COCO datasets, respectively.
Revised: Table 1 and Tablein Appendix D show the classiﬁcation scores for CCP and COCO datasets, respectively.

Original: Our method outperforms CycleGAN in all classiﬁcation experiments, e.g., ours achieves 23.2% accuracy for the pants → shorts task, while CycleGAN obtains only 8.5%.
Revised: Our method outperforms CycleGAN in all classiﬁcation experiments, e.g., ours achieves 23.2% accuracy for the pants → shorts task, while CycleGAN obtains only 8.5%.

Original: 3. A BLATION S TUDY
Revised: 3. A BLATION S TUDY

Original: We now investigate the effects of each component of our proposed method in Figure 9.
Revised: We now investigate the effects of each component of our proposed method in Figure 9.

Original: Our method is composed of the InstaGAN architecture, the context preserving loss L ctx , and the sequential minibatch inference/training technique.
Revised: Our method is composed of the InstaGAN architecture, the context preserving loss L ctx , and the sequential minibatch inference/training technique.

Original: We progressively add each component to the baseline model,
Revised: We progressively add each component to the baseline model,

Original: CycleGAN (with doubled parameters).
Revised: CycleGAN (with doubled parameters).

Original: First, we study the effect of our architecture.
Revised: First, we study the effect of our architecture.

Original: For fair comparison, we train a CycleGAN model with an additional input channel, which translates the mask-augmented image, hence we call it CycleGAN+Seg.
Revised: For fair comparison, we train a CycleGAN model with an additional input channel, which translates the mask-augmented image, hence we call it CycleGAN+Seg.

Original: Unlike our architecture which translates the set of instance masks, CycleGAN+Seg translates the union of all masks at once.
Revised: Unlike our architecture which translates the set of instance masks, CycleGAN+Seg translates the union of all masks at once.

Original: Due to this,
Revised: Due to this,

Original: CycleGAN+Seg fails to translate some instances and often merge them.
Revised: CycleGAN+Seg fails to translate some instances and often merge them.

Original: On the other hand, our architecture keeps every instance and disentangles better.
Revised: On the other hand, our architecture keeps every instance and disentangles better.

Original: Second, we study the effect of the context preserving loss: it not only preserves the background better (row 2), but also improves the translation results as it regularizes the mapping (row 3).
Revised: Second, we study the effect of the context preserving loss: it not only preserves the background better (row 2), but also improves the translation results as it regularizes the mapping (row 3).

Original: Third, we study the effect of our sequential translation: it not only improves the generalization performance (row 2,3) but also improves the translation results on few instances, via data augmentation (row 1).
Revised: Third, we study the effect of our sequential translation: it not only improves the generalization performance (row 2,3) but also improves the translation results on few instances, via data augmentation (row 1).

Original: Finally, Figure 10 reports how much the sequential translation, denoted by “Seq”, is effective in inference and training, compared to the one-step approach, denoted by “One”.
Revised: Finally, Figure 10 reports how much the sequential translation, denoted by “Seq”, is effective in inference and training, compared to the one-step approach, denoted by “One”.

Original: For the one-step training, we consider only two instances, as it is the maximum number affordable for our machines.
Revised: For the one-step training, we consider only two instances, as it is the maximum number affordable for our machines.

Original: On the other hand, for the sequential training, we sequentially train two instances twice, i.e., images of four instances.
Revised: On the other hand, for the sequential training, we sequentially train two instances twice, i.e., images of four instances.

Original: For the one-step inference, we translate the entire set at once, and for the sequential inference, we sequentially translate two instances at each iteration.
Revised: For the one-step inference, we translate the entire set at once, and for the sequential inference, we sequentially translate two instances at each iteration.

Original: We ﬁnd that our sequential algorithm is effective for both training and inference: (a) training/inference =
Revised: We ﬁnd that our sequential algorithm is effective for both training and inference: (a) training/inference =

Original: One/Seq shows blurry results as intermediate data have not shown during training and stacks noise as the iteration goes, and (b) Seq/One shows poor generalization performance for multiple instances as the one-step inference for many instances is not shown in training (due to a limited GPU memory).
Revised: One/Seq shows blurry results as intermediate data have not shown during training and stacks noise as the iteration goes, and (b) Seq/One shows poor generalization performance for multiple instances as the one-step inference for many instances is not shown in training (due to a limited GPU memory).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed a novel method incorporating the set of instance attributes for image-to-image translation.
Revised: We have proposed a novel method incorporating the set of instance attributes for image-to-image translation.

Original: The experiments on different datasets have shown successful image-to-image translation on the challenging tasks of multi-instance transﬁguration, including new tasks, e.g., translating jeans to skirt in fashion images.
Revised: The experiments on different datasets have shown successful image-to-image translation on the challenging tasks of multi-instance transﬁguration, including new tasks, e.g., translating jeans to skirt in fashion images.

Original: We remark that our ideas utilizing the set-structured side information have potential to be applied to other cross-domain generations tasks, e.g., neural machine translation or video generation.
Revised: We remark that our ideas utilizing the set-structured side information have potential to be applied to other cross-domain generations tasks, e.g., neural machine translation or video generation.

Original: Investigating new tasks and new information could be an interesting research direction in the future.
Revised: Investigating new tasks and new information could be an interesting research direction in the future.

Original: A A RCHITECTURE D ETAILS
Revised: A CKNOWLEDGMENTS

Revised: This work was supported by the National Research Council of Science & Technology (NST) grant by the Korea government (MSIP)

Revised: (No. CRC-15-05-ETRI), by the ICT R&D program of

Revised: MSIT/IITP [2016-0-00563, Research on Adaptive Machine Learning Technology Development for

Revised: Intelligent Autonomous Digital Companion], and also by Basic Science Research Program (NRF2017R1E1A1A01077999) through the National Research Foundation of Korea (NRF) funded by the

Revised: Ministry of Science, ICT.

Revised: A A RCHITECTURE D ETAILS

Original: We adopted the network architectures of CycleGAN (Zhu et al., 2017) as the building blocks for our proposed model.
Revised: We adopted the network architectures of CycleGAN (Zhu et al., 2017) as the building blocks for our proposed model.

Original: In speciﬁc, we adopted ResNet 9 -blocks generator (Johnson et al., 2016;
Revised: In speciﬁc, we adopted ResNet 9 -blocks generator (Johnson et al., 2016;

Original: He et al., 2016) and PatchGAN (Isola et al., 2017) discriminator.
Revised: He et al., 2016) and PatchGAN (Isola et al., 2017) discriminator.

Original: ResNet generator is composed of downsampling blocks, residual blocks, and upsampling blocks.
Revised: ResNet generator is composed of downsampling blocks, residual blocks, and upsampling blocks.

Original: We used downsampling blocks and residual blocks for encoders, and used upsampling blocks for generators.
Revised: We used downsampling blocks and residual blocks for encoders, and used upsampling blocks for generators.

Original: On the other hand, PatchGAN discriminator is composed of 5 convolutional layers, including normalization and non-linearity layers.
Revised: On the other hand, PatchGAN discriminator is composed of 5 convolutional layers, including normalization and non-linearity layers.

Original: We used the ﬁrst 3 convolution layers for feature extractors, and the last 2 convolution layers for classiﬁer.
Revised: We used the ﬁrst 3 convolution layers for feature extractors, and the last 2 convolution layers for classiﬁer.

Original: We preprocessed instance segmentation as a binary foreground/background mask, hence simply used it as an 1-channel binary image.
Revised: We preprocessed instance segmentation as a binary foreground/background mask, hence simply used it as an 1-channel binary image.

Original: Also, since we concatenated two or three features to generate the ﬁnal outputs, we doubled or tripled the input dimension of those architectures.
Revised: Also, since we concatenated two or three features to generate the ﬁnal outputs, we doubled or tripled the input dimension of those architectures.

Original: Similar to prior works (Johnson et al., 2016; Zhu et al., 2017), we applied Instance Normalization (IN) (Ulyanov & Lempitsky, 2016) for both generators and discriminators.
Revised: Similar to prior works (Johnson et al., 2016; Zhu et al., 2017), we applied Instance Normalization (IN) (Ulyanov & Lempitsky, 2016) for both generators and discriminators.

Original: In addition, we observed that applying Spectral Normalization (SN) (Miyato et al., 2018) for discriminators signiﬁcantly improves the performance, although we used LSGAN (Mao et al., 2017), while the original motivation of SN was to enforce Lipschitz condition to match with the theory of WGAN (Arjovsky et al., 2017; Gulrajani et al., 2017).
Revised: In addition, we observed that applying Spectral Normalization (SN) (Miyato et al., 2018) for discriminators signiﬁcantly improves the performance, although we used LSGAN (Mao et al., 2017), while the original motivation of SN was to enforce Lipschitz condition to match with the theory of WGAN (Arjovsky et al., 2017; Gulrajani et al., 2017).

Original: We also applied SN for generators as suggested in Self-Attention GAN (Zhang et al., 2018), but did not observed gain for our setting.
Revised: We also applied SN for generators as suggested in Self-Attention GAN (Zhang et al., 2018), but did not observed gain for our setting.

Original: B T RAINING D ETAILS
Revised: B T RAINING D ETAILS

Original: For all the experiments, we simply set λ cyc = 10 , λ idt = 10 , and λ ctx = 10 for our loss (7).
Revised: For all the experiments, we simply set λ cyc = 10 , λ idt = 10 , and λ ctx = 10 for our loss (7).

Original: We used Adam (Kingma & Ba, 2014) optimizer with batch size 4, training with 4 GPUs in parallel.
Revised: We used Adam (Kingma & Ba, 2014) optimizer with batch size 4, training with 4 GPUs in parallel.

Original: All networks were trained from scratch, with learning rate of 0.0002 for G and 0.0001 for D , and β 1 = 0 .
Revised: All networks were trained from scratch, with learning rate of 0.0002 for G and 0.0001 for D , and β 1 = 0 .

Original: 5 , β 2 = 0 .
Revised: 5 , β 2 = 0 .

Original: 999 for the optimizer.
Revised: 999 for the optimizer.

Original: Similar to CycleGAN (Zhu et al., 2017), we kept learning rate for ﬁrst 100 epochs and linearly decayed to zero for next 100 epochs for multi-human parsing (MHP) (Zhao et al., 2018) and COCO (Lin et al., 2014) dataset, and kept learning rate for ﬁrstepochs and linearly decayed for next 200 epochs for clothing co-parsing (CCP)
Revised: Similar to CycleGAN (Zhu et al., 2017), we kept learning rate for ﬁrst 100 epochs and linearly decayed to zero for next 100 epochs for multi-human parsing (MHP) (Zhao et al., 2018) and COCO (Lin et al., 2014) dataset, and kept learning rate for ﬁrstepochs and linearly decayed for next 200 epochs for clothing co-parsing (CCP)

Original: (Yang et al., 2014) dataset, as it contains smaller number of samples.
Revised: (Yang et al., 2014) dataset, as it contains smaller number of samples.

Original: We sampled two classes from the datasets above, and used it as two domains for translation.
Revised: We sampled two classes from the datasets above, and used it as two domains for translation.

Original: We resized images with size 300 × 200 (height × width) for CCP dataset, 240 × 160 for MHP dataset, and 200 × 200 for COCO dataset, respectively.
Revised: We resized images with size 300 × 200 (height × width) for CCP dataset, 240 × 160 for MHP dataset, and 200 × 200 for COCO dataset, respectively.

Original: C T REND OF T RANSLATION R ESULTS
Revised: C T REND OF T RANSLATION R ESULTS

Original: We tracked the trend of translation results over epoch increases, as shown in Figure 11.
Revised: We tracked the trend of translation results over epoch increases, as shown in Figure 11.

Original: Both image and mask smoothly adopted to the target instances.
Revised: Both image and mask smoothly adopted to the target instances.

Original: For example, the remaining parts in legs slowly disappears, and the skirt slowly constructs the triangular shapes.
Revised: For example, the remaining parts in legs slowly disappears, and the skirt slowly constructs the triangular shapes.

Original: D Q UANTITATIVE R ESULTS
Revised: D Q UANTITATIVE R ESULTS

Original: We evaluated the classiﬁcation score for CCP and COCO dataset.
Revised: We evaluated the classiﬁcation score for CCP and COCO dataset.

Original: Unlike CCP dataset, COCO dataset suffers from the false positive problem, that the classiﬁer fails to determine if the generator produced target instances on the right place.
Revised: Unlike CCP dataset, COCO dataset suffers from the false positive problem, that the classiﬁer fails to determine if the generator produced target instances on the right place.

Original: To overcome this issue, we measured the masked classiﬁcation score, where the input images are masked by the corresponding segmentations.
Revised: To overcome this issue, we measured the masked classiﬁcation score, where the input images are masked by the corresponding segmentations.

Original: We note that CycleGAN and our method showed comparable results for the na¨ıve classiﬁcation score, but ours outperformed for the masked classiﬁcation score, as it reduces the false positive problem
Revised: We note that CycleGAN and our method showed comparable results for the na¨ıve classiﬁcation score, but ours outperformed for the masked classiﬁcation score, as it reduces the false positive problem

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E M ORE T RANSLATION R ESULTS
Revised: E M ORE T RANSLATION R ESULTS

Original: We present more qualitative results in high resolution images.
Revised: We present more qualitative results in high resolution images.

Original: F M ORE C OMPARISONS WITH C YCLE GAN+S EG
Revised: F M ORE C OMPARISONS WITH C YCLE GAN+S EG

Original: To demonstrate the effectiveness of our method further, we provide more comparison results with
Revised: To demonstrate the effectiveness of our method further, we provide more comparison results with

Original: CycleGAN+Seg.
Revised: CycleGAN+Seg.

Original: Since CycleGAN+Seg translates all instances at once, it often (a) fails to translate instances, or (b) merges multiple instances (see Figure 23 and 25), or (c) generates multiple instances from one instance (see Figure 24 and 26).
Revised: Since CycleGAN+Seg translates all instances at once, it often (a) fails to translate instances, or (b) merges multiple instances (see Figure 23 and 25), or (c) generates multiple instances from one instance (see Figure 24 and 26).

Original: On the other hand, our method does not have such issues due to its instance-aware nature.
Revised: On the other hand, our method does not have such issues due to its instance-aware nature.

Original: In addition, since the unioned mask losses the original shape information, our instance-aware method produces better shape results ( e.g. , see row 1 of Figure 25).
Revised: In addition, since the unioned mask losses the original shape information, our instance-aware method produces better shape results ( e.g. , see row 1 of Figure 25).

Original: G G ENERALIZATION OF T RANSLATED M ASKS
Revised: G G ENERALIZATION OF T RANSLATED M ASKS

Original: To show that our model generalizes well, we searched the nearest training neighbors (in L 2 -norm) of translated target masks.
Revised: To show that our model generalizes well, we searched the nearest training neighbors (in L 2 -norm) of translated target masks.

Original: As reported in Figure 27, we observe that the translated masks (col 3,4) are often much different from the nearest neighbors (col 5,6).
Revised: As reported in Figure 27, we observe that the translated masks (col 3,4) are often much different from the nearest neighbors (col 5,6).

Original: This conﬁrms that our model does not simply memorize training instance masks, but learns a mapping that generalizes for target instances.
Revised: This conﬁrms that our model does not simply memorize training instance masks, but learns a mapping that generalizes for target instances.

Original: H T RANSLATION R ESULTS OF C ROP & A TTACH B ASELINE
Revised: H T RANSLATION R ESULTS OF C ROP & A TTACH B ASELINE

Original: For interested readers, we also present the translation results of the simple crop & attach baseline in Figure 28, that ﬁnd the nearest neighbors of the original masks from target masks, and crop & attach the corresponding image to the original image.
Revised: For interested readers, we also present the translation results of the simple crop & attach baseline in Figure 28, that ﬁnd the nearest neighbors of the original masks from target masks, and crop & attach the corresponding image to the original image.

Original: Here, since the distance in pixel space ( e.g. ,
Revised: Here, since the distance in pixel space ( e.g. ,

Original: L 2 -norm) obviously does not capture semantics, the cropped instances do not ﬁt with the original contexts as well.
Revised: L 2 -norm) obviously does not capture semantics, the cropped instances do not ﬁt with the original contexts as well.

Original: I V IDEO T RANSLATION R ESULTS
Revised: I V IDEO T RANSLATION R ESULTS

Original: For interested readers, we also present video translation results in Figure 29.
Revised: For interested readers, we also present video translation results in Figure 29.

Original: Here, we use a predicted segmentation (generated by a pix2pix (Isola et al., 2017) model as in Figure 8 and Figure 12) for each frame.
Revised: Here, we use a predicted segmentation (generated by a pix2pix (Isola et al., 2017) model as in Figure 8 and Figure 12) for each frame.

Original: Similar to CycleGAN, our method shows temporally coherent results, even though we did not used any explicit regularization.
Revised: Similar to CycleGAN, our method shows temporally coherent results, even though we did not used any explicit regularization.

Original: One might design a more advanced version of our model utilizing temporal patterns e.g. , using the idea of Recycle-GAN (Bansal et al., 2018) for video-to-video translation, which we think is an interesting future direction to explore.
Revised: One might design a more advanced version of our model utilizing temporal patterns e.g. , using the idea of Recycle-GAN (Bansal et al., 2018) for video-to-video translation, which we think is an interesting future direction to explore.

Original: J R ECONSTRUCTION R ESULTS
Revised: J R ECONSTRUCTION R ESULTS

Original: For interested readers, we also report the translation and reconstruction results of our method in
Revised: For interested readers, we also report the translation and reconstruction results of our method in

Original: Figure 30.
Revised: Figure 30.

Original: One can observe that our method shows good reconstruction results while showing good translation results.
Revised: One can observe that our method shows good reconstruction results while showing good translation results.

Original: This implies that our translated results preserve the original context well.
Revised: This implies that our translated results preserve the original context well.