Original: R ETHINKING THE T RULY U NSUPERVISED I MAGE - TOI MAGE T RANSLATION
Revised: R ETHINKING THE T RULY U NSUPERVISED I MAGE - TOI MAGE T RANSLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Every recent image-to-image translation model uses either image-level ( i.e. inputoutput pairs) or set-level ( i.e. domain labels) supervision at a minimum.
Revised: Every recent image-to-image translation model uses either image-level ( i.e. inputoutput pairs) or set-level ( i.e. domain labels) supervision at a minimum.

Original: However, even the set-level supervision can be a serious bottleneck for data collection in practice.
Revised: However, even the set-level supervision can be a serious bottleneck for data collection in practice.

Original: In this paper, we tackle image-to-image translation in a fully unsupervised setting, i.e. , neither paired images nor domain labels.
Revised: In this paper, we tackle image-to-image translation in a fully unsupervised setting, i.e. , neither paired images nor domain labels.

Original: To this end, we propose a truly unsupervised image-to-image translation model (TUNIT) that simultaneously learns to separate image domains and translate input images into the estimated domains.
Revised: To this end, we propose a truly unsupervised image-to-image translation model (TUNIT) that simultaneously learns to separate image domains and translate input images into the estimated domains.

Original: Experimental results show that our model achieves comparable or even better performance than the  supervised model trained with full labels, generalizes well on various datasets, and is robust against the choice of hyperparameters ( e.g. the preset number of pseudo domains).
Revised: Experimental results show that our model achieves comparable or even better performance than the set-level supervised model trained with full labels, generalizes well on various datasets, and is robust against the choice of hyperparameters ( e.g. the preset number of pseudo domains).

Original: In addition, TUNIT extends well to the semi-supervised scenario with various amount of labels provided.
Revised: In addition, TUNIT extends well to the semi-supervised scenario with various amount of labels provided.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Given an image of one domain, image-to-image translation is a task to generate the plausible images of the other domains.
Revised: Given an image of one domain, image-to-image translation is a task to generate the plausible images of the other domains.

Original: Based on the success of conditional generative models (Mirza & Osindero, 2014; Sohn et al., 2015), many image translation methods have been proposed either using imagelevel supervision ( e.g. paired data) (Isola et al., 2017; Hoffman et al., 2018; Zhu et al., 2017b; Wang et al., 2018; Park et al., 2019) or using set-level supervision ( e.g. domain labels) (Zhu et al., 2017a;
Revised: Based on the success of conditional generative models (Mirza & Osindero, 2014; Sohn et al., 2015), many image translation methods have been proposed either using imagelevel supervision ( e.g. paired data) (Isola et al., 2017; Hoffman et al., 2018; Zhu et al., 2017b; Wang et al., 2018; Park et al., 2019) or using set-level supervision ( e.g. domain labels) (Zhu et al., 2017a;

Original: Kim et al., 2017; Liu et al., 2017; Huang et al., 2018; Liu et al., 2019; Lee et al., 2020).
Revised: Kim et al., 2017; Liu et al., 2017; Huang et al., 2018; Liu et al., 2019; Lee et al., 2020).

Original: Though the latter approach is generally called unsupervised as a counterpart of the former, it actually assumes that the domain labels are given a priori .
Revised: Though the latter approach is generally called unsupervised as a counterpart of the former, it actually assumes that the domain labels are given a priori .

Original: This assumption can be a serious bottleneck in practice as the number of domains and samples increases.
Revised: This assumption can be a serious bottleneck in practice as the number of domains and samples increases.

Original: For example, labeling individual samples of a large dataset, such as FFHQ, is expensive, and the distinction across domains can be ambiguous.
Revised: For example, labeling individual samples of a large dataset, such as FFHQ, is expensive, and the distinction across domains can be ambiguous.

Original: Here, we ﬁrst clarify that unsupervised image-to-image translation should strictly denote the task without any supervision neither paired images nor domain labels.
Revised: Here, we ﬁrst clarify that unsupervised image-to-image translation should strictly denote the task without any supervision neither paired images nor domain labels.

Original: Under this deﬁnition, our goal is to develop an unsupervised translation model given a mixed set of images of many domains (Figure 1).
Revised: Under this deﬁnition, our goal is to develop an unsupervised translation model given a mixed set of images of many domains (Figure 1).

Original: We tackle this problem by formulating three sub-problems: 1) clustering the images by approximating the set-level characteristics ( i.e. domains), 2) encoding the individual content and style of an input image, and 3) learning a mapping function among the estimated domains.
Revised: We tackle this problem by formulating three sub-problems: 1) clustering the images by approximating the set-level characteristics ( i.e. domains), 2) encoding the individual content and style of an input image, and 3) learning a mapping function among the estimated domains.

Original: To this end, we introduce a guiding network that simultaneously solves 1) unsupervised domain classiﬁcation and 2) style encoding.
Revised: To this end, we introduce a guiding network that simultaneously solves 1) unsupervised domain classiﬁcation and 2) style encoding.

Original: It has two branches of providing pseudo domain labels and encoding style features, which are later used in the discriminator and the generator training, respectively.
Revised: It has two branches of providing pseudo domain labels and encoding style features, which are later used in the discriminator and the generator training, respectively.

Original: We employ a differentiable clustering method based on mutual information maximization for estimating domain labels.
Revised: We employ a differentiable clustering method based on mutual information maximization for estimating domain labels.

Original: This helps the guiding network group similar images together while evenly separate their categories.
Revised: This helps the guiding network group similar images together while evenly separate their categories.

Original: For embedding style codes, we adopt a contrastive loss (Hadsell et al., 2006;
Revised: For embedding style codes, we adopt a contrastive loss (Hadsell et al., 2006;

Original: He et al., 2020; Chen et al., 2020a), which leads the model to further understand the dissimilarity between images, resulting in better representation learning.
Revised: He et al., 2020; Chen et al., 2020a), which leads the model to further understand the dissimilarity between images, resulting in better representation learning.

Original: Finally, conditioned on the style features and domain labels from the guiding network, we use generative adversarial networks (GAN) to learn the image translation functions across various domains.
Revised: Finally, conditioned on the style features and domain labels from the guiding network, we use generative adversarial networks (GAN) to learn the image translation functions across various domains.

Original: Although GAN and the guiding network play different roles, we do not separate their training process– our guiding network participates in the translation process.
Revised: Although GAN and the guiding network play different roles, we do not separate their training process– our guiding network participates in the translation process.

Original: By doing so, the guiding network can exploit gradients from GAN training.
Revised: By doing so, the guiding network can exploit gradients from GAN training.

Original: The guiding network now understands the recipes of
Revised: The guiding network now understands the recipes of

Original: [Figure]
Revised: [Figure]

Original: domain-separating attributes because the generator wants the style code tocontain
Revised: domain-separating attributes because the generator wants the style code to

Original: - mation to fool the domain-speciﬁc discriminator, and vice versa.
Revised: - mation to fool the domain-speciﬁc discriminator, and vice versa.

Original: Thanks to this interaction between the guiding network and GAN, our model successfully separates domains and translates images.
Revised: Thanks to this interaction between the guiding network and GAN, our model successfully separates domains and translates images.

Original: We quantitatively and qualitatively compare our model with the existing set-level supervised method under unsupervised and semi-supervised setting.
Revised: We quantitatively and qualitatively compare our model with the existing set-level supervised method under unsupervised and semi-supervised setting.

Original: The experiments on various datasets show that the proposed model outperforms the previous method over all different levels of supervision.
Revised: The experiments on various datasets show that the proposed model outperforms the previous method over all different levels of supervision.

Original: Our experimental results show that, by exploiting the synergy between two tasks, the guiding network helps the image translation model to largely improve the generation performance.
Revised: Our experimental results show that, by exploiting the synergy between two tasks, the guiding network helps the image translation model to largely improve the generation performance.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: • We clarify the deﬁnition of unsupervised image-to-image translation and to the best of our knowledge, our model is the ﬁrst to succeed in this task in an end-to-end manner.
Revised: • We clarify the deﬁnition of unsupervised image-to-image translation and to the best of our knowledge, our model is the ﬁrst to succeed in this task in an end-to-end manner.

Original: • We propose the guiding network to handle the unsupervised translation task and show that the interaction between translation and clustering is helpful for the task.
Revised: • We propose the guiding network to handle the unsupervised translation task and show that the interaction between translation and clustering is helpful for the task.

Original: • We show the effectiveness of our model through the extensive experiments on various datasets.
Revised: • We show the effectiveness of our model through the extensive experiments on various datasets.

Original: • We conﬁrm that our model is applicable to various numbers of clusters and the practical case, where ground truth labels of several samples are available.
Revised: • We conﬁrm that our model is applicable to various numbers of clusters and the practical case, where ground truth labels of several samples are available.

Original:  T RULY U NSUPERVISED I MAGE - TO -I MAGE T RANSLATION (TUNIT)
Revised:  T RULY U NSUPERVISED I MAGE - TO -I MAGE T RANSLATION (TUNIT)

Original: We consider the unsupervised image-to-image translation problem, where we have images χ from
Revised: We consider the unsupervised image-to-image translation problem, where we have images χ from

Original: K domains ( K ≥ 2 ) without domain labels y .
Revised: K domains ( K ≥ 2 ) without domain labels y .

Original: Here, K is an unknown property of the dataset.
Revised: Here, K is an unknown property of the dataset.

Revised: Throughout the paper, we denote K as the actual number of domains in a dataset and ˆ K as the arbitrarily chosen number of domains to train models.

Original: We design a module that integrates both a domain classiﬁer and a style encoder, which we call guiding network.
Revised: We design a module that integrates both a domain classiﬁer and a style encoder, which we call guiding network.

Original: It guides the translation by feeding reference images as the style code to the generator and as the pseudo domain labels to the discriminator.
Revised: It guides the translation by feeding reference images as the style code to the generator and as the pseudo domain labels to the discriminator.

Original: Using the feedback from the discriminator regarding the pseudo labels, the generator synthesizes images of the target domains ( e.g. breeds) while respecting styles ( e.g. fur patterns) of the reference images and maintaining the content ( e.g. pose) of source images (Figure 2).
Revised: Using the feedback from the discriminator regarding the pseudo labels, the generator synthesizes images of the target domains ( e.g. breeds) while respecting styles ( e.g. fur patterns) of the reference images and maintaining the content ( e.g. pose) of source images (Figure 2).

Original: 2. L EARNING TO PRODUCE DOMAIN LABELS AND ENCODE STYLE FEATURES
Revised: 2. L EARNING TO PRODUCE DOMAIN LABELS AND ENCODE STYLE FEATURES

Original: In our framework, the guiding network E plays a central role as an unsupervised domain classiﬁer as well as a style encoder.
Revised: In our framework, the guiding network E plays a central role as an unsupervised domain classiﬁer as well as a style encoder.

Original: Our guiding network E consists of two branches, E C and E S , each of which learns to provide domain labels and style codes, respectively.
Revised: Our guiding network E consists of two branches, E C and E S , each of which learns to provide domain labels and style codes, respectively.

Original: In experiments, we compare our guiding network against straightforward approaches, i.e. ., K-means on image or feature space.
Revised: In experiments, we compare our guiding network against straightforward approaches, i.e. ., K-means on image or feature space.

Original: Unsupervised domain classiﬁcation.
Revised: Unsupervised domain classiﬁcation.

Original: The discriminator requires target domain labels to provide useful gradients for image translation into the target domain.
Revised: The discriminator requires target domain labels to provide useful gradients for image translation into the target domain.

Original: E C adopts a differentiable clusteringtechnique to provide pseudo domain labels of reference images, maximizing the mutual information
Revised: E C adopts a differentiable clustering

Original: E
Revised: E

Original: G
Revised: G

Original: D D E
Revised: D D E

Original: Input image
Revised: Input image

Original: R / F R / F
Revised: R / F R / F

Original: R / F
Revised: R / F

Original: Argmax + Detach Argmax + Detach
Revised: Argmax + Detach Argmax + Detach

Original: Input image
Revised: Input image

Original: Reference image
Revised: Reference image

Original: Generated image Generated image
Revised: Generated image Generated image

Original: P s eudo l abe l
Revised: P s eudo l abe l

Original: S t y l e c ode
Revised: S t y l e c ode

Original: P s eudo l abe l
Revised: P s eudo l abe l

Original: S t y l e c ode
Revised: S t y l e c ode

Original: Source image
Revised: Source image

Original: R / F R / F
Revised: R / F R / F

Original: R / F
Revised: R / F

Original: [Figure]
Revised: [Figure]

Original:  (MI) between an image x and its randomly augmented version x + (Ji et al., 2019).
Revised: technique to provide pseudo domain labels of reference images, maximizing the mutual information (MI) between an image x and its randomly augmented version x + (Ji et al., 2019).

Original: The optimum of the mutual information I ( p , p + ) is reached as the entropy H ( p ) is maximum and the conditional entropy H ( p | p + ) is minimum, where p = E C ( x ) represents the softmax output from E C , indicating a probability vector of x over  K domains.
Revised: The optimum of the mutual information I ( p , p + ) is reached as the entropy H ( p ) is maximum and the conditional entropy H ( p | p + ) is minimum, where p = E C ( x ) represents the softmax output from E C , indicating a probability vector of x over ˆ K domains.

Revised: Please refer to Section 3.3 for more details about ˆ K .

Original: Maximizing MI encourages E C to assign the same domain label to the pair ( x and x + ) while evenly distributing entire samples to all domains.
Revised: Maximizing MI encourages E C to assign the same domain label to the pair ( x and x + ) while evenly distributing entire samples to all domains.

Original: Formally, E C maximizes the mutual information:
Revised: Formally, E C maximizes the mutual information:

Original: [Equation]
Revised: [Equation]

Original: where P = E x + ∼ f ( x ) | x ∼ p data ( x ) [ E C ( x
Revised: where P [Equation]

Original: [Equation]
Revised: [Equation]

Original: (1) where f is a composition of random augmentations such as random cropping and afﬁne transformation.
Revised: where f is a composition of random augmentations such as random cropping and afﬁne transformation.

Original: P i
Revised: P i

Original: = P ( p = i ) denotes the  K -dimensional marginal probability vector, and P ij = P ( p
Revised: = P ( p = i ) denotes the ˆ K -dimensional marginal probability vector, and P ij = P ( p

Original: = i, p + = j ) denotes the joint probability.
Revised: = i, p + = j ) denotes the joint probability.

Original: To provide a deterministic one-hot label to the discriminator, we use the argmax operation (i.e. y = argmax ( E C ( x )) ).
Revised: To provide a deterministic one-hot label to the discriminator, we use the argmax operation (i.e. y = argmax ( E C ( x )) ).

Original: We note that the mutual information is one way to implement TUNIT, therefore, any differentiable clustering methods can be adopted such as SCAN (Van Gansbeke et al., 2020).
Revised: We note that the mutual information is one way to implement TUNIT, therefore, any differentiable clustering methods can be adopted such as SCAN (Van Gansbeke et al., 2020).

Original: Style encoding and improving domain classiﬁcation.
Revised: Style encoding and improving domain classiﬁcation.

Original: E S encodes an image into a style code s which provides translation guide for the generator.
Revised: E S encodes an image into a style code s which provides translation guide for the generator.

Original: In addition to the style guide to the generator,
Revised: In addition to the style guide to the generator,

Original: E S is beneﬁcial in improving unsupervised domain classiﬁcation, where pseudo labels from E C fail to scale up when samples are complex and diverse ( e.g. , AnimalFaces (Liu et al., 2019)).
Revised: E S is beneﬁcial in improving unsupervised domain classiﬁcation, where pseudo labels from E C fail to scale up when samples are complex and diverse ( e.g. , AnimalFaces (Liu et al., 2019)).

Original: Since E S is an another branch of the guiding network, imposing the contrastive loss (He et al., 2020) on the style codes improves representation of the shared embeddings:
Revised: Since E S is an another branch of the guiding network, imposing the contrastive loss (He et al., 2020) on the style codes improves representation of the shared embeddings:

Original: [Equation]
Revised: [Equation]

Original: where s = E S ( x ) .
Revised: where s = E S ( x ) .

Revised: x and x + denote an image and randomly augmented version of x , respectively.

Original: This ( N +1) -way
Revised: This ( N +1) -way

Original: classiﬁcation enables E to utilize not only the similarity of the positive pair ( s , s + ) but also the dissimilarity of the negative pairs ( s , s − i ), where the negative style codes s − i are stored into a queue from the previously sampled images.
Revised: classiﬁcation enables E to utilize not only the similarity of the positive pair ( s , s + ) but also the dissimilarity of the negative pairs ( s , s − i ), where the negative style codes s − i are stored into a queue from the previously sampled images.

Original: We observe that adding this objective signiﬁcantly improves unsupervised classiﬁcation accuracy on AnimalFaces from 68.0% to 84.1% compared to the previous approach (Ji et al., 2019).
Revised: We observe that adding this objective signiﬁcantly improves unsupervised classiﬁcation accuracy on AnimalFaces from 68.0% to 84.1% compared to the previous approach (Ji et al., 2019).

Original: 2. L EARNING TO TRANSLATE IMAGES WITH THE DOMAIN GUIDANCE
Revised: 2. L EARNING TO TRANSLATE IMAGES WITH THE DOMAIN GUIDANCE

Original: In this subsection, we describe how to perform the unsupervised image-to-image translation under the guidance of our guiding network.
Revised: In this subsection, we describe how to perform the unsupervised image-to-image translation under the guidance of our guiding network.

Original: For successful translation, the model should provide the realistic images containing the visual feature of the target domain.
Revised: For successful translation, the model should provide the realistic images containing the visual feature of the target domain.

Original: To this end, we adopt three losses: 1) adversarial loss to produce realistic images, 2) style contrastive loss that encourages the model not to ignore the style codes, 3) image reconstruction loss for preserving the domain-invariant features.
Revised: To this end, we adopt three losses: 1) adversarial loss to produce realistic images, 2) style contrastive loss that encourages the model not to ignore the style codes, 3) image reconstruction loss for preserving the domain-invariant features.

Original: We explain each loss and the overall objective for each network.
Revised: We explain each loss and the overall objective for each network.

Original: Adversarial loss.
Revised: Adversarial loss.

Original: For adversarial training, we adopt a variant of conditional discriminator, the multitask discriminator (Mescheder et al., 2018).
Revised: For adversarial training, we adopt a variant of conditional discriminator, the multitask discriminator (Mescheder et al., 2018).

Original: It is designed to conduct discrimination for each domain simultaneously.
Revised: It is designed to conduct discrimination for each domain simultaneously.

Original: However, its gradient is calculated only with the loss for estimating the domain of the input image.
Revised: However, its gradient is calculated only with the loss for estimating the domain of the input image.

Original: For the domain label of the input image, we utilize the pseudo label from the guiding network.
Revised: For the domain label of the input image, we utilize the pseudo label from the guiding network.

Original: Formally, given the pseudo label ˜ y for a reference image ˜ x , we train our generator G and multi-task discriminator D via the adversarial loss:
Revised: Formally, given the pseudo label ˜ y for a reference image ˜ x , we train our generator G and multi-task discriminator D via the adversarial loss:

Original: [Equation]
Revised: [Equation]

Original: where D ˜ y
Revised: where D ˜ y

Original: ( · ) denotes the logit from the domain-speciﬁc ( ˜ y ) discriminator, and ˜ s =
Revised: ( · ) denotes the logit from the domain-speciﬁc ( ˜ y ) discriminator, and ˜ s =

Original: E S
Revised: E S

Original: (˜ x ) denotes a target style code of the reference image ˜ x .
Revised: (˜ x ) denotes a target style code of the reference image ˜ x .

Original: The generator G learns to translate x to the target domain ˜
Revised: The generator G learns to translate x to the target domain ˜

Original: y while reﬂecting the style code ˜ s .
Revised: y while reﬂecting the style code ˜ s .

Original: Style constrastive loss.
Revised: Style constrastive loss.

Original: In order to prevent a degenerate case where the generator ignores the given style code ˜ s and synthesize a random image of the domain ˜ y , we impose a style contrastive loss:
Revised: In order to prevent a degenerate case where the generator ignores the given style code ˜ s and synthesize a random image of the domain ˜ y , we impose a style contrastive loss:

Original: [Equation]
Revised: [Equation]

Original: Here, s (cid:48) =
Revised: Here, s (cid:48) =

Original: E S ( G ( x , ˜ s )) denotes the style code of the translated image G ( x , ˜ s ) and s − i denotes the negative style codes, which are from the same queue used in  (2).
Revised: E S ( G ( x , ˜ s )) denotes the style code of the translated image G ( x , ˜ s ) and s − i denotes the negative style codes, which are from the same queue used in equation (2).

Revised: And we follow the training scheme of MoCo (He et al., 2020) as equation (2).

Original: The above loss guides the generated image G ( x , ˜ s ) to have a style similar to the reference image ˜ x and dissimilar to negative (other) samples.
Revised: The above loss guides the generated image G ( x , ˜ s ) to have a style similar to the reference image ˜ x and dissimilar to negative (other) samples.

Original: By doing so, we also avoid the degenerated solution where the encoder maps all the images to the same style code of the reconstruction loss (Choi et al., 2020) based on L1 or L2 norm.
Revised: By doing so, we also avoid the degenerated solution where the encoder maps all the images to the same style code of the reconstruction loss (Choi et al., 2020) based on L1 or L2 norm.

Original: Image reconstruction loss.
Revised: Image reconstruction loss.

Original: To ensure that the generator G can reconstruct the source image x when given with its original style s = E S ( x ) , we impose an image reconstruction loss:
Revised: To ensure that the generator G can reconstruct the source image x when given with its original style s = E S ( x ) , we impose an image reconstruction loss:

Original: [Equation]
Revised: [Equation]

Original: This objective not only ensures the generator G to preserve domain-invariant characteristics (e.g., pose) of its input image x , but also helps to learn the style representation of the guiding network E by extracting the original style s of the source image x .
Revised: This objective not only ensures the generator G to preserve domain-invariant characteristics (e.g., pose) of its input image x , but also helps to learn the style representation of the guiding network E by extracting the original style s of the source image x .

Original: Overall objective.
Revised: Overall objective.

Original: Finally, we train the three networks jointly as follows:
Revised: Finally, we train the three networks jointly as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ ’s are hyperparameters.
Revised: where λ ’s are hyperparameters.

Original: Note that our guiding network E receives feedback from L G , which is essential for our method.
Revised: Note that our guiding network E receives feedback from L G , which is essential for our method.

Original: We discuss the effect of feedback to E on performance in Section 3.1.
Revised: We discuss the effect of feedback to E on performance in Section 3.1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We ﬁrst evaluate TUNIT on labeled datasets by treating them as unlabeled because the desired behaviours of the translation models in labeled datasets are well deﬁned (Section 3.1).
Revised: We ﬁrst evaluate TUNIT on labeled datasets by treating them as unlabeled because the desired behaviours of the translation models in labeled datasets are well deﬁned (Section 3.1).

Original: Here, we provide an ablation study to analyze the effect of each component and compare the models both quantitatively and qualitatively.
Revised: Here, we provide an ablation study to analyze the effect of each component and compare the models both quantitatively and qualitatively.

Original: We then move on to unlabeled datasets to validate our model in the unsupervised scenario in the wild (Section 3.2).
Revised: We then move on to unlabeled datasets to validate our model in the unsupervised scenario in the wild (Section 3.2).

Original: Lastly, we show that TUNIT is robust against the choice of hyperparameters (e.g. the preset number of clusters, K ) and extends well to the semi-supervised scenario (Section 3.3).
Revised: Lastly, we show that TUNIT is robust against the choice of hyperparameters (e.g. the preset number of clusters, K ) and extends well to the semi-supervised scenario (Section 3.3).

Original: In all experiments, we use FUNIT (Liu et al., 2019) as our baseline.
Revised: In all experiments, we use FUNIT (Liu et al., 2019) as our baseline.

Original: Datasets.

Original: For the labeled datasets, we randomly select ten classes among 149 classes of AnimalFaces and 101 classes of Food-101, which we call AnimalFaces-10 and Food-10, respectively.

Original: Here, the labels are used only for the evaluation purpose.

Original: For the unlabeled datasets, we use AFHQ, FFHQ, and LSUN Car (Choi et al., 2020;

Original: Karras et al., 2019; Yu et al., 2015), which do not have any or are missing with ﬁne-grained labels.

Original: Speciﬁcally, AFHQ roughly has three groups (i.e., dog, cat and

Original: [Table] [Table]
Revised: [Table] [Table]

Original: type of food) of the reference images while A does not.
Revised: type of food) of the reference images while A does not.

Original: The model C performs much worse than A and B in that it overly adopts the source image, not adequately merging styles and contents from both sides.
Revised: The model C performs much worse than A and B in that it overly adopts the source image, not adequately merging styles and contents from both sides.

Original: The model D generates more plausible images than C but fails to reﬂect the characteristics of the reference images.
Revised: The model D generates more plausible images than C but fails to reﬂect the characteristics of the reference images.

Original: For example, D on ﬁfth row does not look like several pieces of dumpling due to its shape and dish color, meaning that the reference styles are not properly reﬂected.
Revised: For example, D on ﬁfth row does not look like several pieces of dumpling due to its shape and dish color, meaning that the reference styles are not properly reﬂected.

Original: Similarly, E also fails to generate the dumpling in the ﬁfth row.
Revised: Similarly, E also fails to generate the dumpling in the ﬁfth row.

Original: TUNIT with sequential training F reﬂects the visual features of each reference on both datasets.
Revised: TUNIT with sequential training F reﬂects the visual features of each reference on both datasets.

Original: However, in terms of visual ﬁdelity, we observe that G consistently outperforms F .
Revised: However, in terms of visual ﬁdelity, we observe that G consistently outperforms F .

Original: Akin to the quantitative results, TUNIT achieves equivalent or even better visual quality than the  supervised model A and B . wild), but each group contains diverse species and these species labels are not provided.
Revised: Akin to the quantitative results, TUNIT achieves equivalent or even better visual quality than the set-level supervised model A and B . Datasets. For the labeled datasets, we randomly select ten classes among 149 classes of AnimalFaces and 101 classes of Food-101, which we call AnimalFaces-10 and Food-10, respectively.

Revised: Here, the labels are used only for the evaluation purpose.

Original:  FFHQ and
Revised: For the unlabeled datasets, we use AFHQ, FFHQ, and LSUN Car (Choi et al., 2020; Karras et al., 2019; Yu et al., 2015), which do not have any or are missing with ﬁne-grained labels. Speciﬁcally, AFHQ roughly has three groups (i.e., dog, cat and wild), but each group contains diverse species and these species labels are not provided. FFHQ and

Original: LSUN Car contain various human faces and cars without any labels, respectively.
Revised: LSUN Car contain various human faces and cars without any labels, respectively.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: We report two scores to assess the generated images.
Revised: We report two scores to assess the generated images.

Original: First, to provide a general sense of image quality, we use the mean of class-wise Fr´enchet Inception Distance (mFID) (Heusel et al., 2017).
Revised: First, to provide a general sense of image quality, we use the mean of class-wise Fr´enchet Inception Distance (mFID) (Heusel et al., 2017).

Original: It can avoid the degenerate case of the original FID, which assigns a good score whenthe model conveying the source image as is.
Revised: It can avoid the degenerate case of the original FID, which assigns a good score when

Original: Additionally, to provide a ﬁner assessment of the generated images, we report Density and Coverage (D&C) (Naeem et al., 2020).

Original: D&C separately evaluates the ﬁdelity and the diversity of the model outputs, which is also known to be robust against outliers and model hyperparameters (e.g. the number of samples used for evaluation).

Original: Denote that a lower mFID score means better image quality, and D&C scores that are bigger or closer to 1.0 indicate the better ﬁdelity and diversity, respectively.

Original: Please refer to Appendix C for the detailed information.

Original: Source Reference
Revised: Source Reference

Original: FUNIT
Revised: FUNIT

Original: TUNIT
Revised: TUNIT

Original: Source Reference
Revised: Source Reference

Original: FUNIT
Revised: FUNIT

Original: TUNIT
Revised: TUNIT

Original: A F
Revised: A F

Original: HQ
Revised: HQ

Original: C A T
Revised: C A T

Original: FF
Revised: FF

Original: HQ
Revised: HQ

Original: A F H
Revised: A F H

Original: Q W
Revised: Q W

Original: I L D
Revised: I L D

Original: L S U
Revised: L S U

Original: N C
Revised: N C

Original: A R
Revised: A R

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: AFHQ Wild does not have ground-truth labels, each  is coloredwith the guiding prediction.
Revised: AFHQ Wild does not have ground-truth labels, each point is colored the guiding prediction.

Original: Although we set the number of domains to be quite large (  K = 10 ), the network separates one species into two domains, which are so closely located that the model creates six clusters
Revised: Although we set the number of domains to be quite large ( ˆ K = 10 ), the network separates one species into two domains, which are so closely located that the model creates six clusters.

Revised: the model conveying the source image as is.

Revised: Additionally, to provide a ﬁner assessment of the generated images, we report Density and Coverage (D&C) (Naeem et al., 2020).

Revised: D&C separately evaluates the ﬁdelity and the diversity of the model outputs, which is also known to be robust against outliers and model hyperparameters (e.g. the number of samples used for evaluation).

Revised: Denote that a lower mFID score means better image quality, and D&C scores that are bigger or closer to 1.0 indicate the better ﬁdelity and diversity, respectively.

Revised: Please refer to Appendix C for the detailed information.

Original: 3. C OMPARATIVE E VALUATION ON L ABELED D ATASETS
Revised: 3. C OMPARATIVE E VALUATION ON L ABELED D ATASETS

Original: Table 1 summarizes the effect of each component of TUNIT and rigorous comparisons with the state-of-the-art supervised method, FUNIT.
Revised: Table 1 summarizes the effect of each component of TUNIT and rigorous comparisons with the state-of-the-art supervised method, FUNIT.

Original: First, we report the  supervised performance of FUNIT and its variant (Table 1).
Revised: First, we report the set-level supervised performance of FUNIT and its variant (Table 1).

Original: Here, A is the original FUNIT and B denotes the modiﬁed FUNIT using our architecture ( e.g. We do not use PatchGAN discriminator), which brings a large improvement over every score on both datasets.
Revised: Here, A is the original FUNIT and B denotes the modiﬁed FUNIT using our architecture ( e.g. We do not use PatchGAN discriminator), which brings a large improvement over every score on both datasets.

Original: One simple way to extend B to the unsupervised scenario is to add an off-the-shelf clustering method and use its estimated labels instead of the ground truth.
Revised: One simple way to extend B to the unsupervised [Figure] ˆ K AnimalFaces

Revised: [Table]

Revised: [Figure]

Revised: ˆ K =

Revised: ˆ K

Revised: = ˆ K =

Revised: [Table]

Revised: [Table]

Revised: K = ˆ K = scenario is to add an off-the-shelf clustering method and use its estimated labels instead of the ground truth.

Original: We employ K-means clustering on the image space for C , and the pretrained feature space for D .
Revised: We employ K-means clustering on the image space for C , and the pretrained feature space for D .

Original: Here, we use ResNet-50 (He et al., 2016) features trained with MoCo v2 (Chen et al., 2020b) on ImageNet.
Revised: Here, we use ResNet-50 (He et al., 2016) features trained with MoCo v2 (Chen et al., 2020b) on ImageNet.

Original: Not surprisingly, because the estimated labels are inaccurate, the overall performance signiﬁcantly drops.
Revised: Not surprisingly, because the estimated labels are inaccurate, the overall performance signiﬁcantly drops.

Original: Although using the pretrained features helps a little, not only is it far from the  supervised performance but it requires three steps to train the entire model, which complicates the application.
Revised: Although using the pretrained features helps a little, not only is it far from the setlevel supervised performance but it requires three steps to train the entire model, which complicates the application.

Original: This can be partially addressed by employing the differentiable clustering method (Ji et al., 2019), which trains VGG-11BN (Simonyan & Zisserman, 2015) with mutual information maximization [Figure] K AnimalFaces
Revised: This can be partially addressed by employing the differentiable clustering method (Ji et al., 2019), which trains VGG-11BN (Simonyan & Zisserman, 2015) with mutual information maximization that makes E .

Original: [Table]

Original: Source

Original: Reference

Original: K =

Original: K =

Original: [Table]

Original: =

Original: [Table]

Original: =

Original: K = K = K = that makes E . This reduces the number of training steps from three to two and provides better label estimation, which enables the model to approach the performance of original FUNIT 
Revised: This reduces the number of training steps from three to two and provides better label estimation, which enables the model to approach the performance of original FUNIT A .

Original: However, as seen in the coverage score, the sample diversity is unsatisfactory.
Revised: However, as seen in the coverage score, the sample diversity is unsatisfactory.

Original: Finally, we build TUNIT by introducing the guiding network and the new objective functions described in Section 2.
Revised: Finally, we build TUNIT by introducing the guiding network and the new objective functions described in Section 2.

Original: The changes signiﬁcantly improve the accuracy on both datasets, particularly achieving similar mFID of the improved  supervised model B .
Revised: The changes signiﬁcantly improve the accuracy on both datasets, particularly achieving similar mFID of the improved set-level supervised model B .

Original: Our ﬁnal model, G matches or outperforms mFID and D&C of B .
Revised: Our ﬁnal model, G matches or outperforms mFID and D&C of B .

Original: This is impressive because B utilizes oracles for training while G has no labels.
Revised: This is impressive because B utilizes oracles for training while G has no labels.

Original: Notably, TUNIT can cover 7%p wider support of the data on AnimalFaces-10 than
Revised: Notably, TUNIT can cover 7%p wider support of the data on AnimalFaces-10 than

Original: B .
Revised: B .

Original: We conjecture that TUNIT beneﬁts from the style codes that represent meaningful domain features learned by clustering.
Revised: We conjecture that TUNIT beneﬁts from the style codes that represent meaningful domain features learned by clustering.

Original: Comparing F and G , we conﬁrm that  the joint training of style encoder and clustering improves overall performances.
Revised: By comparing F and G , we conﬁrm that they are comparable in terms of clustering and G is more stable in terms of inter-dataset performance. Therefore, we adopt the joint training of style encoder and clustering as our ﬁnal model ( G ).

Original: In addition, we remove the adversarial loss for training the guiding network.
Revised: In addition, we remove the adversarial loss for training the guiding network.

Original: It directly degrades the performance; mFID changes from 47.7 to 63.0 on AnimalFaces-10.
Revised: It directly degrades the performance; mFID changes from 47.7 to 63.0 on AnimalFaces-10.

Original: It indicates that our training scheme takes an important portion of performance gains.
Revised: It indicates that our training scheme takes an important portion of performance gains.

Original: Qualitative results also shows superiority of TUNIT over competitors Figure 3.
Revised: Qualitative results also show superiority of TUNIT over competitors (Figure 3). [Table] [Table]

Original: 3. V ALIDATION ON U NLABELED D ATASET
Revised: 3. V ALIDATION ON U NLABELED D ATASET

Original: We evaluate TUNIT on the unlabeled datasets having no clear separations of the domains, which are
Revised: We evaluate TUNIT on the unlabeled datasets having no clear separations of the domains, which are

Original: AFHQ, FFHQ and LSUN Cars.
Revised: AFHQ, FFHQ and LSUN Cars.

Original: For AFHQ, we train three individual models for dog , cat and wild .
Revised: For AFHQ, we train three individual models for dog , cat and wild .

Original: For all experiments, FUNIT is used as a baseline.
Revised: For all experiments, FUNIT is used as a baseline.

Original: We train it by presuming all labels to be the same as one.
Revised: We train it by presuming all labels to be the same as one.

Original: We set  K =10 for all the TUNIT models.
Revised: We set ˆ K =10 for all the TUNIT models.

Original: More discussions on  K will be in Section 3.3.
Revised: More discussions on ˆ K will be in Section 3.3.

Original: Figure 4 demonstrates the results.
Revised: Figure 4 demonstrates the results.

Original: We observe that the results of TUNIT adequately reﬂect the style feature of the references such as the textures of cats or cars and the species of the wilds.
Revised: We observe that the results of TUNIT adequately reﬂect the style feature of the references such as the textures of cats or cars and the species of the wilds.

Original: Although
Revised: Although

Original: FFHQ has no clear domain distinctions, TUNIT captures the existence of glasses or smile as do- [Table] [Table] mains, and then add or remove glasses or smile.
Revised: FFHQ has no clear domain distinctions, TUNIT captures the existence of glasses or smile as domains, and then add or remove glasses or smile.

Original: However, FUNIT performs much worse than TUNIT in this truly unsupervised scenario.
Revised: However, FUNIT performs much worse than TUNIT in this truly unsupervised scenario.

Original: For example, FUNIT outputs the inputs as is (cats and wilds) or insufﬁciently reﬂects the species (third row of AFHQ Wild).
Revised: For example, FUNIT outputs the inputs as is (cats and wilds) or insufﬁciently reﬂects the species (third row of AFHQ Wild).

Original: For FFHQ, despite that FUNIT makes some changes, the changes are not interpreted as meaningful domain translations.
Revised: For FFHQ, despite that FUNIT makes some changes, the changes are not interpreted as meaningful domain translations.

Original: For LSUN Car,
Revised: For LSUN Car,

Original: FUNIT fails to keep the ﬁdelity.
Revised: FUNIT fails to keep the ﬁdelity.

Original: We also visualize the style space of both models to qualitatively assess the quality of the representation.
Revised: We also visualize the style space of both models to qualitatively assess the quality of the representation.

Original: Figure 5 shows the t-SNE maps trained on AFHQ Wild and the examples of each cluster.
Revised: Figure 5 shows the t-SNE maps trained on AFHQ Wild and the examples of each cluster.

Original: Surprisingly, TUNIT organizes the samples according to the species where it roughly separates the images into six species.
Revised: Surprisingly, TUNIT organizes the samples according to the species where it roughly separates the images into six species.

Original: Although we set  K to be overly large, the model represents one species into two domains where those two domains position much closely.
Revised: Although we set ˆ K to be overly large, the model represents one species into two domains where those two domains position much closely.

Original: The highly disentangled, meaningful style features can be an important factor in the success of our model.
Revised: The highly disentangled, meaningful style features can be an important factor in the success of our model.

Original: On the other hand, the style features of FUNIT hardly learn meaningful domains so that the model cannot conduct the translation properly as shown in Figure 4.
Revised: On the other hand, the style features of FUNIT hardly learn meaningful domains so that the model cannot conduct the translation properly as shown in Figure 4.

Original: Because of the page limit, we include more results in Appendix E, F.
Revised: Because of the page limit, we include more results in Appendix E, G.

Original:  A NALYSIS ON G ENERALIZABILITY
Revised: 3. A NALYSIS ON G ENERALIZABILITY

Original: Robustness to various  K ’s. When TUNIT conducts clustering for estimating domain labels, the number of clusters  K can affect the overall performances.
Revised: Robustness to various ˆ K ’s. When TUNIT conducts clustering for estimating domain labels, the number of clusters ˆ K can affect the overall performances.

Original: Here, we study the effects on different  K on the labeled datasets and report them in Figure 6 and Table 2.
Revised: Here, we study the effects on different ˆ K on the labeled datasets and report them in Figure 6 and Table 2.

Original: As expected, the model performs best in terms of mFID when  K equals to the ground truth K ( i.e.  K =10).
Revised: As expected, the model performs best in terms of mFID when ˆ K equals to the ground truth K ( i.e. ˆ K =10).

Original: Additionally, TUNIT performs reasonably well for sufﬁciently large  K ( ≥ 7 ) on both datasets.
Revised: Additionally, TUNIT performs reasonably well for sufﬁciently large ˆ K ( ≥ 7 ) on both datasets.

Original: From this study, we conclude that
Revised: From this study, we conclude that

Original: TUNIT is relatively robust against  K as long as it is sufﬁciently large.
Revised: TUNIT is relatively robust against ˆ K as long as it is sufﬁciently large.

Original: With Few labels.
Revised: With Few labels.

Original: We investigate whether or not TUNIT is effective for the partially labeled dataset that corresponds to a more practical scenario.
Revised: We investigate whether or not TUNIT is effective for the partially labeled dataset that corresponds to a more practical scenario.

Original: To this end, we use AnimalFaces-10 and Food-10.
Revised: To this end, we use AnimalFaces-10 and Food-10.

Original: We partition the dataset D into the labeled set D sup and the unlabeled set D un with varying ratio γ =
Revised: We partition the dataset D into the labeled set D sup and the unlabeled set D un with varying ratio γ =

Original: |D sup | / |D| .
Revised: |D sup | / |D| .

Original: Like before, we choose FUNIT as a competitor.
Revised: Like before, we choose FUNIT as a competitor.

Original: We ﬁrst train the networks while changing γ from 0.2 to 1.0 and report the result Table 3.
Revised: We ﬁrst train the networks while changing γ from 0.2 to 1.0 and report the result Table 3.

Original: As γ decreases, the performance of FUNIT signiﬁcantly degrades whereas our model maintains mFID around 45 and 55 on both datasets.
Revised: As γ decreases, the performance of FUNIT signiﬁcantly degrades whereas our model maintains mFID around 45 and 55 on both datasets.

Original: We also train an auxiliary classiﬁer (VGG-11BN) with D sup then provide pseudo-labels for D un to
Revised: We also train an auxiliary classiﬁer (VGG-11BN) with D sup then provide pseudo-labels for D un to

Original: FUNIT.
Revised: FUNIT.

Original: As shown in Table 4, the performance of FUNIT is no longer sensitive to the changes in γ but still much worse than TUNIT.
Revised: As shown in Table 4, the performance of FUNIT is no longer sensitive to the changes in γ but still much worse than TUNIT.

Original: Although semi-supervised learning schemes can further improve
Revised: Although semi-supervised learning schemes can further improve

Original: FUNIT, TUNIT outperforms FUNIT using all labels as seen in Table 1.
Revised: FUNIT, TUNIT outperforms FUNIT using all labels as seen in Table 1.

Original: Under the empirical results in the semi-supervised scenario, we verify that TUNIT is also effective for semi-supervised setting.
Revised: Under the empirical results in the semi-supervised scenario, we verify that TUNIT is also effective for semi-supervised setting.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We argue that the unsupervised image-to-image translation should not utilize any supervision, such as image-level ( i.e. paired) or set-level ( i.e. unpaired) supervision.
Revised: We argue that the unsupervised image-to-image translation should not utilize any supervision, such as image-level ( i e. paired) or set-level ( i e. unpaired) supervision.

Original: Under this rigorous regime, many previous studies fall into the  supervised framework that uses the domain information at a minimum.
Revised: Under this rigorous regime, many previous studies fall into the set-level supervised framework that uses the domain information at a minimum.

Original: In this paper, for the ﬁrst time, we proposed an effective model to handle the truly unsupervised image-to-image translation.
Revised: In this paper, for the ﬁrst time we proposed an effective model to handle the truly unsupervised image-to-image translation.

Original: To this end, we suggested the guiding network that performs unsupervised representation learning for providing pseudo labels and the image translation tasks.
Revised: To this end, we suggested the guiding network that performs unsupervised representation learning for providing pseudo labels and the image translation tasks.

Original: The experimental results showed that guiding network indeed exploits the synergy between two tasks, and the proposed model successfully conducts the unsupervised-image-to-image translation.
Revised: The experimental results showed that guiding network indeed exploits the synergy between two tasks, and the proposed model successfully conducts the unsupervised-image-to-image translation.

Original: We also showed the generalizability on the value of  K and the partially labeled scenario.
Revised: We also showed the generalizability on the value of ˆ K and the partially labeled scenario.

Original: A PPENDIX A R ELATED WORK
Revised: A PPENDIX A R ELATED WORK

Original: Image-to-image translation.
Revised: Image-to-image translation.

Original: Since the seminal work of Pix2Pix (Isola et al., 2017), image-to-image translation models have shown impressive results (Zhu et al., 2017a; Liu et al., 2017; Kim et al., 2017; Kupyn et al., 2018;
Revised: Since the seminal work of Pix2Pix (Isola et al., 2017), image-to-image translation models have shown impressive results (Zhu et al., 2017a; Liu et al., 2017; Kim et al., 2017; Kupyn et al., 2018;

Original: Choi et al., 2018; Huang et al., 2018;
Revised: Choi et al., 2018; Huang et al., 2018;

Original: Liu et al., 2019; Yang et al., 2019).
Revised: Liu et al., 2019; Yang et al., 2019).

Original: Exploiting the cycle consistency constraint, these methods were able to train the model with a setlevel supervision (domains) solely.
Revised: Exploiting the cycle consistency constraint, these methods were able to train the model with a setlevel supervision (domains) solely.

Original: However, acquiring domain information can be a huge burden in practical applications where a large amount of data are gathered from several mixed domains, e.g. , web images (Yalniz et al., 2019).
Revised: However, acquiring domain information can be a huge burden in practical applications where a large amount of data are gathered from several mixed domains, e.g. , web images (Yalniz et al., 2019).

Original: Not only does this complicates the data collection, but it restricts the methods only applicable to the existing dataset and domains.
Revised: Not only does this complicates the data collection, but it restricts the methods only applicable to the existing dataset and domains.

Original: Inspired from few shot learning,
Revised: Inspired from few shot learning,

Original: Liu et al.
Revised: Liu et al.

Original: (2019) proposed FUNIT that works on previously unseen target classes.
Revised: (2019) proposed FUNIT that works on previously unseen target classes.

Original: However, FUNIT still requires the labels for training.
Revised: However, FUNIT still requires the labels for training.

Original: Recently, Bahng et al.
Revised: Recently, Bahng et al.

Original: (Bahng et al., 2020) has partially addressed this by adopting the ImageNet pre-trained classiﬁer for extracting domain information.
Revised: (Bahng et al., 2020) has partially addressed this by adopting the ImageNet pre-trained classiﬁer for extracting domain information.

Original: Unlike the previous methods, we aim to design an image-to-image translation model that can be applied without any supervision such as a pre-trained network or supervision on both the train and the test datasets.
Revised: Unlike the previous methods, we aim to design an image-to-image translation model that can be applied without any supervision such as a pre-trained network or supervision on both the train and the test datasets.

Original: Unsupervised representation learning and clustering.
Revised: Unsupervised representation learning and clustering.

Original: Unsupervised representation learning aims to extract meaningful features for downstream tasks without any human supervision.
Revised: Unsupervised representation learning aims to extract meaningful features for downstream tasks without any human supervision.

Original: To this end, many researchers have proposed to utilize the information that can be acquired from the data itself (Gidaris et al., 2018;
Revised: To this end, many researchers have proposed to utilize the information that can be acquired from the data itself (Gidaris et al., 2018;

Original: Hjelm et al., 2019; Ji et al., 2019;
Revised: Hjelm et al., 2019; Ji et al., 2019;

Original: He et al., 2020; Van Gansbeke et al., 2020).
Revised: He et al., 2020; Van Gansbeke et al., 2020).

Original: Recently, by incorporating the contrastive learning into a dictionary learning framework,
Revised: Recently, by incorporating the contrastive learning into a dictionary learning framework,

Original: MoCo (He et al., 2020) has achieved outstanding performance in various downstream tasks under reasonable mini-batch size.
Revised: MoCo (He et al., 2020) has achieved outstanding performance in various downstream tasks under reasonable mini-batch size.

Original: On the other hand, IIC (Ji et al., 2019) have utilized the mutual information maximization in a unsupervised way so that the network clusters images while assigning the images evenly.
Revised: On the other hand, IIC (Ji et al., 2019) have utilized the mutual information maximization in a unsupervised way so that the network clusters images while assigning the images evenly.

Original: Though IIC provided a principled way to perform unsupervised clustering, it fails to scale up when combined with a difﬁcult downstream task such as image-to-image translation.
Revised: Though IIC provided a principled way to perform unsupervised clustering, it fails to scale up when combined with a difﬁcult downstream task such as image-to-image translation.

Original: By taking the best of both worlds, we aim to solve unsupervised image-to-image translation.
Revised: By taking the best of both worlds, we aim to solve unsupervised image-to-image translation.

Original: B T RAINING DETAILS
Revised: B T RAINING DETAILS

Original: We train the guiding network for the ﬁrst 65K iterations while freezing the update from both the generator and the discriminator.
Revised: We train the guiding network for the ﬁrst 65K iterations while freezing the update from both the generator and the discriminator.

Original: Then, we train the whole framework 100K more iterations for training all the networks.
Revised: Then, we train the whole framework 100K more iterations for training all the networks.

Original: The batch size is set to 32 and 16 for 128 × 128 and 256 × 256 images, respectively.
Revised: The batch size is set to 32 and 16 for 128 × 128 and 256 × 256 images, respectively.

Original: Training takes about 36 hours on a single Tesla V100 GPU with our implementation using PyTorch(Paszke et al., 2017).
Revised: Training takes about 36 hours on a single Tesla V100 GPU with our implementation using PyTorch(Paszke et al., 2017).

Original: We use Adam (Kingma & Ba, 2014) optimizer with β 1 = 0 .
Revised: We use Adam (Kingma & Ba, 2014) optimizer with β 1 = 0 .

Original: 9 , β 2 = 0 .for
Revised: 9 , β 2 = 0 .for

Original: the guiding network, and RMSprop (Hinton et al., 2012) optimizer with α = 0 .
Revised: the guiding network, and RMSprop (Hinton et al., 2012) optimizer with α = 0 .

Original: 99 for the generator and the discriminator.
Revised: 99 for the generator and the discriminator.

Original: All learning rates are set to 0 .
Revised: All learning rates are set to 0 .

Original: 0001 with a weight decay 0 . 0001 .
Revised: 0001 with a weight decay 0 . 0001 .

Original: We adopt hinge version adversarial loss (Lim & Ye, 2017; Tran et al., 2017) with R 1 regularization (Mescheder et al., 2018) using γ = 10 (Eq. 5).
Revised: We adopt hinge version adversarial loss (Lim & Ye, 2017; Tran et al., 2017) with R 1 regularization (Mescheder et al., 2018) using γ = 10 (Eq. 5).

Original: We set λ rec = 0 .
Revised: We set λ rec = 0 .

Original: 1 , λ G style = 0 .
Revised: 1 , λ G style = 0 .

Original: 01 , λ E style = 1 , and λ MI = 5 in equation.
Revised: 01 , λ E style = 1 , and λ MI = 5 in equation.

Original: 6 for all experiments.
Revised: 6 for all experiments.

Original: When the guiding network is simultaneously trained with the generator, we decrease λ E style and λ MI to 0.1 and 0.5, respectively.
Revised: When the guiding network is simultaneously trained with the generator, we decrease λ E style and λ MI to 0.1 and 0.5, respectively.

Original: For evaluation, we use the exponential moving average over the parameters (Karras et al., 2018) of the guiding network and the generator.
Revised: For evaluation, we use the exponential moving average over the parameters (Karras et al., 2018) of the guiding network and the generator.

Original: We initialize the weights of convolution layers with He initialization (He et al., 2015), all biases to zero, and weights of linear layers from N (0 , 0 . 01) with zero biases.
Revised: We initialize the weights of convolution layers with He initialization (He et al., 2015), all biases to zero, and weights of linear layers from N (0 , 0 . 01) with zero biases.

Original: The source code will be available publicly.
Revised: The source code will be available publicly.

Original: C E VALUATION PROTOCOL
Revised: C E VALUATION PROTOCOL

Original: For evaluation, we use class-wise Fr´echet Inception Distance (FID) (Heusel et al., 2017), which is often called mFID in literatures and D&C (Naeem et al., 2020).
Revised: For evaluation, we use class-wise Fr´echet Inception Distance (FID) (Heusel et al., 2017), which is often called mFID in literatures and D&C (Naeem et al., 2020).

Original: FID measures Fr´echet distance between real and fake samples embedded by the last average pooling layer of Inception-V3 pretrained on ImageNet.
Revised: FID measures Fr´echet distance between real and fake samples embedded by the last average pooling layer of Inception-V3 pretrained on ImageNet.

Original: Class-wise FID is obtained by averaging the FIDs of individual classes.
Revised: Class-wise FID is obtained by averaging the FIDs of individual classes.

Original: In the experiments with fewer labels, we report the mean value of best ﬁve mFID’s
Revised: In the experiments with fewer labels, we report the mean value of best ﬁve mFID’s

Original: over 100K iterations.
Revised: over 100K iterations.

Original: For example, we use entire real images of each class and generate 810 fake images where 18 ×
Revised: For example, we use entire real images of each class and generate 810 fake images where 18 ×

Original: ( K − 1) source images ( K = 10 for AnimalFaces-10) and ﬁve reference images of AnimalFaces10 are used to produce those fake images.
Revised: ( K − 1) source images ( K = 10 for AnimalFaces-10) and ﬁve reference images of AnimalFaces10 are used to produce those fake images.

Original: We choose the source images from all classes except for the target class.
Revised: We choose the source images from all classes except for the target class.

Original: For each source image, the ﬁve references are selected arbitrarily.
Revised: For each source image, the ﬁve references are selected arbitrarily.

Original: For D&C, we generate fake images the similar number of training images with randomly selected source and reference images.
Revised: For D&C, we generate fake images the similar number of training images with randomly selected source and reference images.

Original: Then, we use Inception-V3 pre-trained on ImageNet for extracting feature vectors and measure D&C by using the feature vectors.
Revised: Then, we use Inception-V3 pre-trained on ImageNet for extracting feature vectors and measure D&C by using the feature vectors.

Original: D A RCHITECTURE DETAILS
Revised: D A RCHITECTURE DETAILS

Original: For the guiding network, we use VGG11 before the linear layers followed by the average pooling operation as the shared part and append two branches E class and E style .
Revised: For the guiding network, we use VGG11 before the linear layers followed by the average pooling operation as the shared part and append two branches E class and E style .

Original: The branches are one linear layer with  K and 128 dimensional outputs, respectively.
Revised: The branches are one linear layer with ˆ K and 128 dimensional outputs, respectively.

Original: The detailed information of the generator, the guiding network and the discriminator architectures are provided in Table 5, Table 6 and Table 7. 
Revised: The detailed information of the generator, the guiding network and the discriminator architectures are provided in Table 5, Table 6 and Table 7. [Table] AYER

Original: [Table]
Revised: [Table]

Original: R ESAMPLE
Revised: ESAMPLE

Original: NORM O UTPUT S HAPE Image x - -
Revised: N

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Conv

Original: [Table]
Revised: [Table]

Original: IN
Revised: IN

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ch
Revised: ch

Original: Conv
Revised: Conv

Original: [Table]
Revised: [Table]

Original: Stride IN 64 × 64 × 2 ch
Revised: IN 64 × 64 × [Table] ch

Original: Conv4 ×
Revised: Conv [Table]

Original: Stride
Revised: Stride

Original: IN 32 × 32 × 4 ch
Revised: IN [Table] [Table] ch

Original: Conv
Revised: Conv

Original: [Table]
Revised: [Table]

Original: Stride
Revised: Stride

Original: IN 16 × 16 × 8 ch
Revised: IN 16 × 16 × [Table]

Original:  ResBlkIN 16 × 16 × 8 ch ResBlkIN16 ×
Revised: ch ResBlkIN 16 × 16 × 8 ch ResBlkIN

Original: [Table] 8 ch
Revised: [Table] [Table] ch

Original: ResBlkAdaIN 16 × 16 × 8 ch
Revised: ResBlkAdaIN [Table] [Table]

Original:  ResBlkAdaIN 16 × 16 ×8 ch
Revised: ch ResBlkAdaIN 16 × 16 ×

Revised: [Table]

Original: Conv5 ×
Revised: ch Conv [Table]

Original: Upsample
Revised: Upsample

Original: AdaIN 32 ×
Revised: AdaIN [Table]

Original: [Table]
Revised: [Table]

Original: 4 ch
Revised: ch

Original: Conv
Revised: Conv

Original: [Table]
Revised: [Table]

Original: Upsample
Revised: Upsample

Original: AdaIN 64 × 64 × 2 ch
Revised: AdaIN 64 × 64 × [Table] ch

Original: Conv
Revised: Conv

Original: [Table]
Revised: [Table]

Original: Upsample
Revised: Upsample

Original: AdaIN
Revised: AdaIN

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Conv
Revised: ch

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E T -SNE VISUALIZATION & CLUSTER EXAMPLE IMAGES E. AFHQ C AT
Revised: E T -SNE VISUALIZATION & CLUSTER EXAMPLE IMAGES E. AFHQ C AT

Original: [Figure]
Revised: [Figure]

Original: E.
Revised: E.

Original: [Figure]
Revised: [Figure]

Original: OG
Revised: OG

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: F A DDITIONAL C OMPARISON WITH FUNIT: AFHQ, LSUN C AR AND FFHQ

Original: [Figure]
Revised: [Figure]

Revised: I

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: n

Original: [Figure]
Revised: [Figure]

Original: F 
Revised: F UN I T T UN I T

Original: AFHQ, LSUN C AR , FFHQ, A NIMAL F ACES -10, AND S2W
Revised: S ou r ce R e f e r e n ce F UN I T

Original: F.
Revised: T UN I T S ou r ce R e f e r e n ce

Original: A NIMAL
Revised: F UN I T T UN I T

Revised: G A DDITIONAL R ESULTS OF TUNIT: AFHQ, LSUN C AR , FFHQ, A NIMAL F ACES -10, AND S2W

Revised: G

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: G FID AND LPIPS ON UNLABELED DATASET
Revised: H FID AND LPIPS ON UNLABELED DATASET

Original: [Figure]
Revised: [Figure]

Original: We also utilize LPIPS to evaluate the models in addition to FID and D&C.
Revised: We also utilize LPIPS to evaluate the models in addition to FID and D&C.

Original: However, LPIPS is not proper to evaluate the loyalty for reﬂecting the reference image and the ﬁdelity of images, we use
Revised: However, LPIPS is not proper to evaluate the loyalty for reﬂecting the reference image and the ﬁdelity of images, we use

Original: LPIPS with FID.
Revised: LPIPS with FID.

Original: Figure 18 shows the result.
Revised: Figure 22 shows the result.

Original: It is clear that a model with high FID and LPIPS generates a noise-like image.
Revised: It is clear that a model with high FID and LPIPS generates a noise-like image.

Original: Even if FID is low, a model with high LPIPS also fails to conduct the reference-guided image translation, because it does not preserve the structure of the source image.
Revised: Even if FID is low, a model with high LPIPS also fails to conduct the reference-guided image translation, because it does not preserve the structure of the source image.

Original: The model with low LPIPS and high FID might be an adversarial example of LPIPS.
Revised: The model with low LPIPS and high FID might be an adversarial example of LPIPS.

Original: We generate the image via optimization on LPIPS.
Revised: We generate the image via optimization on LPIPS.

Original: If a model exhibits low FID and LPIPS, it might not reﬂect the visual feature of the reference image enough.
Revised: If a model exhibits low FID and LPIPS, it might not reﬂect the visual feature of the reference image enough.

Original: The simple combination of LPIPS and FID can detect several failed models but can not evaluate the loyalty for the reference image.
Revised: The simple combination of LPIPS and FID can detect several failed models but can not evaluate the loyalty for the reference image.

Original: We suggest that the rigorous way to combine several metrics for the quantitative evaluation of the reference-guided translation might be a interesting future work.
Revised: We suggest that the rigorous way to combine several metrics for the quantitative evaluation of the reference-guided translation might be a interesting future work.