M INIMAL G EOMETRY -D ISTORTION C ONSTRAINT FOR U NSUPERVISED I MAGE - TO -I MAGE T RANSLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervised image-to-image (I2I) translation, which aims to learn a domain mapping function without paired data, is very challenging because the function is highly under-constrained.
Despite the signiﬁcant progress in constraining the mapping function, current methods suffer from the geometry distortion problem: the geometry structure of the translated image is inconsistent with the input source image, which may cause the undesired distortions in the translated images.
To remedy this issue, we propose a novel I2I translation constraint, called Minimal
Geometry-Distortion Constraint (MGC), which promotes the consistency of geometry structures and reduce the unwanted distortions in translation by reducing the randomness of color transformation in the translation process.
To facilitate estimation and maximization of MGC, we propose an approximate representation of mutual information called relative Squared-loss Mutual Information (rSMI) that can be efﬁciently estimated analytically.
We demonstrate the effectiveness of our MGC by providing quantitative and qualitative comparisons with the state-of-the-art methods on several benchmark datasets.
I NTRODUCTION
Image-to-image translation, or domain mapping, aims to translate an image in the source domain X to the target domain Y .
It has been extensively studied (Pathak et al., 2016; Isola et al., 2017; Liu et al., 2019) and has been applied to various vision tasks (Sela et al., 2017; Siddiquee et al., 2019;
Ghosh et al., 2019; Tomei et al., 2019; Wu et al., 2019).
Early works considered supervised image-toimage (I2I) translation, where paired samples { ( x i , y i ) } Ni =1 drawn from the joint distribution P XY are available.
In the presence of paired data, methods based on conditional generative adversarial networks can generate high-quality translations (Isola et al., 2017;
Wang et al., 2018; Pathak et al., 2016).
However, since paired data are often unavailable or expensive to obtain, unsupervised I2I translation has attracted intense attention in recent years (Zhu et al., 2017; Yi et al., 2017;
Kim et al., 2017; Benaim & Wolf, 2017; Huang et al., 2018; Lee et al., 2019; Kim et al., 2019; Park et al., 2020).
Beneﬁting from generative adversarial networks (GANs) (Goodfellow et al., 2014), one can perform unsupervised I2I translation by ﬁnding G XY such that the translated images and target domain images have similar distributions, i.e. , P G XY ( X ) ≈ P Y .
Due to an inﬁnite number of functions that can satisfy the adversarial loss, GAN alone cannot guarantee the learning of the true mapping function, resulting in sub-optimal translation performance.
To remedy this issue, various kinds of constraints have been placed on the learned mapping function.
For instance, the well-known cycle-consistency (Zhu et al., 2017; Kim et al., 2017; Yi et al., 2017) enforces the translation function G XY to be bijective.
DistanceGAN (Benaim & Wolf, 2017) preserves the pairwise distances in the source images.
GcGAN (Fu et al., 2019) forces the function to be smooth w.r.t.
certain geometric transformations of input images.
DRIT++ (Lee et al., 2019) and MUNIT (Huang et al., 2018) learn disentangled representations by embedding images onto a domain-invariant content space and a domain-speciﬁc attribute space and the mapping function can be derived from representation learning components.
However, the mapping functions learned by current methods are still far from satisfactory in real applications.
Here we consider a simple but widely applicable image translation task, i.e. , geometryinvariant translation task.
In this task, the geometric structure (e.g. the shapes of objects) in images in the source and target domain is invariant and the variation of photometric information of a certain geometric area is expected to conform with the change of style information, such as the colour of a leaf is green in summer and white in winter.
Existing methods enforced in geometry-invariant translation [Figure] tasks still suffer from geometry distortion problem, where the geometry structures in the source and translated image are not consistent, resulting in the mismatch of input and translated images.
A representative example is that the mapping function G XY learned with the cycle-consistency (Zhu et al., 2017) often changes the geometry structures of digits in the SVHN → MNIST translation task, so some digits in source domain images are translated accidentally into other digits.
The geometry distortion problem hinders the application of unsupervised geometry-invariant translation methods into a wide range of computer vision applications, such as domain adaptation (Hoffman et al., 2017), segmentation (Zhu et al., 2017) and style transfer (Huang et al., 2018).
In this paper, we propose a new constraint for unsupervised geometry-invariant image translation, called minimal geometry-distortion constraint (MGC), as a general I2I translation constraint to guarantee the consistency of geometry structure of source and translated images, and thus reduce translation mismatch in the translation process.
We observe that the pixel values before and after translation are usually highly correlated if the geometric structure is preserved because the color transformation is more regular within speciﬁc object regions.
Taking the color transformation of a leaf as an example, the transformation of a green leaf into a red leaf contain less randomness than into a colorful one.
Based on this observation, we propose a mutual information (MI)-based dependency measure that models the nonlinear relationships of pixel values in the source and translated images.
To estimate MI from data, we propose the relative Squared-Loss Mutual Information (rSMI) which can be efﬁciently estimated in an analytic form.
By maximizing rSMI together with the GAN loss, our approach can signiﬁcantly reduce the geometry distortion by better preserving geometric structures.
In the experiments, we incorporate our minimal geometry-distortion constraint into the GAN framework and show its effectiveness of preserving geometric structures when used both independently and combined with existing constraints ( e.g. cycle consistency) to show its compatibility.
The quantitative and qualitative comparisons with baselines (models without MGC) and state-of-the-art methods on several datasets demonstrate the superiority of the proposed MGC constraint.
R ELATED W ORK
Unsupervised Image-to-Image Translation.
In unsupervised image-to-image (I2I) translation, unaligned examples drawn individually from the marginal distribution of the source domain and target domain are available.
Although the subject has obtained some promising progress in recent years, only several works study it from an optimization perspective.
Speciﬁcally, Cyclic consistency based GAN, e.g., CycleGAN (Zhu et al., 2017), DualGAN
(Yi et al., 2017) and DiscoGAN (Kim et al., 2017), is a general approach for this problem.
DistanceGAN (Benaim & Wolf, 2017) and GcGAN (Fu et al., 2019) further introduced distance and geometry transformation consistency to constraint the search space of mapping functions.
Instead of exploiting general constraints for the subject, more works developed novel frameworks to investigate special settings of unsupervised I2I translation.
Several other works (Huang et al., 2018; Choi et al., 2018; Lee et al., 2019; 2018;
Shen et al., 2019) mapped the content and style information of images into disentangled spaces for multimodal translations.
However, we ﬁnd that the complex neural networks and many hyper-parameters make the optimization process unstable (Kim et al., 2019).
In addition, (Dosovitskiy & Brox, 2016;
Johnson et al., 2016; Mechrez et al., 2018; Katzir et al., 2019) tried to reduce the perceptual loss or content loss based on a pre-trained VGG model to reduce the content of two domain image, which is computationally cost and cannot be easily adapted to the data on hand. Further, Wu et al. Katzir et al. (2019) give efforts to change the geometry of source images using geometry information extracted by cVAE (Esser et al., 2018; Kingma & Welling, 2013) or pre-trained VGG network.
However, for many image translation applications that need the geometry structures preservation, e.g. domain adaptation, unsupervised segmentation, geometry-invariant image translation, our method provides a simple way to achieve the goal of preserving geometry-structure.
Mutual Information (MI). Mutual information is a fundamental measure of dependency between two random variables, and it is widely used in machine learning and is particularly suitable for canonical tasks such as multi-modalities images registration (Zitova & Flusser, 2003; Maes et al., 1997; Luan et al., 2008).
Since computing MI is difﬁcult (Paninski, 2003), researchers have taken much effort to improve the estimation of MI.
For example, early works studied Non-parametric models based on Kernel Density Estimator (KDE) (Jonsson & Felsberg, 2006;
Krishnamurthy et al., 2014; Kandasamy et al., 2015; Singh & Póczos, 2014a;b), K-nearest Neighbor Method (KNN) (Kraskov et al., 2004; Kozachenko & Leonenko, 1987), and likelihood-ratio estimator (Suzuki et al., 2008) for MI estimation.
Subsequent works improved the performance in more complicated cases such as discrete-continuous mixtures (Moon et al., 2017; Gao et al., 2017) and segmentation (Zhao et al., 2019).
Recently, MINE (Belghazi et al., 2018; Hjelm et al., 2018) showed that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks.
M ETHODOLOGY
Unsupervised I2I translation aims to ﬁnd a mapping function G XY between two domains X and
Y given unpaired samples { x i } Ni =1 and { y j } Mj =1 drawn from the marginal distributions P X and P Y , respectively.
To solve geometry-distortion problem, we ﬁrst present our motivation of placing the MIbased minimal geometry-distortion constraint, and then give the details about our proposed minimal geometry-distortion constraint (MGC), which aims to reduce the randomness of color transform in the translation process and thus promote the consistency of geometry structure between source and translated images.
3. M OTIVATION
[Figure]
which is estimated by our rSMI method.
As illustrated in Figure 2, 4 (a), and 5, advanced methods ( e.g. CycleGAN, CUT (Park et al., 2020),
Contexual loss (Mechrez et al., 2018), U-GAT-IT (Kim et al., 2019), MUNIT (Huang et al., 2018) can change the geometry structure of input images and potentially cause the mismatch between input and translated images.
Therefore, it is essential to enforce a constraint such that we can ensure the learned function G XY change the image style with minimal geometry distortion.
Our work is the ﬁrst to explore such constraints for unsupervised image-to-image translation.
To reveal the reason for the geometry-distortion in I2I translation, we plot the corresponding pixel values of images before and after translation at the bottom row of Figure 2.
We can see that the pixel values in the translated image (Y-axis) are less dependent of the pixel values (X-axis) in the input images using previous methods ( e.g. CycleGAN, CUT, Contextual loss).
As such, one color within an object region can be randomly mapped to various colors after translation.
A geometric area in a single image is normally occupied by a certain color, so the randomness of the color transformation in the translation process using previous methods distorts the geometry structure of the input image.
Therefore, reducing the randomness of color transformation is a way to alleviate the geometry-distortion problem in current I2I translation methods.
Motivated by the analysis, we develop the minimal geometry-distortion constraint (MGC) as a general and effective constraint to preserve the pixel-level structure during the translation process.
MGC exploits mutual information to model the non-linear dependencies of pixel values between the input and translated images, thus reducing the randomness of color transformation in the translation.
As illustrated in Figure 3, our MGC is enforced into the input and translated images and thus allows onesided unsupervised domain mapping, i.e., G XY can be trained independently from G Y X .
Applying our MGC to a vanilla GAN, the pixel values before and after translation have stronger dependency (higher MI), and thus better preserves the geometric structures as shown in Figure 2.
In the following, we present the details of our approach.
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] +1wzasXYEUI1d7diOiSaUOsCKrkQ/MWXl0nzrOp7Vf/uvFK7yuMowhEcwyn4cAE1uIU6NIDCIzzDK7whhV7QO/qYtxZQPnMIf4A+fwBuMo8F</latexit> MG Constraint
CycleConstraint
CycleConstraint
3. A PPROXIMATE R EPRESENTATION OF M UTUAL I NFORMATION
For a source domain image x i ∈ X and its translation ˆ y i =
G XY ( x i ) , we denote V x i and V
ˆ y i as the random variables for pixels in x i and ˆ y i , respectively.
Thus, pixels in x i , i.e., { v x i j } Mj =1 , can be regarded as data sampled from P V xi , and the pixels in ˆ y i , i.e., { v ˆ y i j } Mj =1 , can be considered as data sampled from P V ˆ yi , where M is the number of pixels of the image.
Formally, the mutual information between V x i and V
ˆ y
i is
[Equation]
where P ( V xi ,V ˆ yi ) is the joint distribution of V x i and V ˆ y i , P V xi ⊗ P V ˆ yi is the product of two marginal distributions P V xi and P V ˆ yi .
Because V x i and V ˆ y i are low-dimensional, a straightforward way to estimate (1) is to estimate the distributions P based on the histogram of the images.
Next, we will introduce how we estimate the mutual information between pixels from two domain images and backpropagate it to optimize parameters in the translation network.
To enable efﬁcient backpropagation, we propose the relative Squared-loss Mutual Information (rSMI), which is an extension of the well-known Squared-loss Mutual Information (SMI) (Suzuki et al., 2009).
For conventional presentation, we denote P V xi ⊗ P V ˆ yi as S i , and P ( V xi ,V ˆ yi ) as Q i .
Then, the SMI based on Pearson (PE) Divergence (Sugiyama et al., 2012) between P V xi and P V y (cid:48)
i is expressed as:
[Equation]
Because S i Q i is unbounded, SMI ( V x i , V ˆ y i ) can be inﬁnity, causing numeric instability in the backpropagation.
We thus use the relative Pearson(rPE) Divergence (Yamada et al., 2013) to alleviate the problem:
[Equation]
Here, we introduce the mixture distribution βS i + (1 − β ) Q i , β ∈ (0 , 1) , to replace Q i .
Beneﬁting from the modiﬁcation, the density ratio will be bounded to [0, 1 β ].
Thus, the proposed rSMI between
V x i and V ˆ y i can be written as:
[Equation]
To estimate the rSMI ( V x i , V ˆ y i ) , we directly estimate the density ratio using a linear combination of kernel functions of { v x i j } M j =1 and { v ˆ y i j } M j =1 :
[Equation]
where φ ∈ R m is the kernel function, α ∈ R m is the parameter vector we need to solve, and m is the number of kernels.
Referring to the least-squares density-difference estimation (Sugiyama et al., 2013), the solved optimal solution of ˆ α is (the derivation is given in the appendix A.1 ):
[Equation]
where R is a positive semi-deﬁnite regularization matrix, n is the sample number, 1 n is the ndimensional vector ﬁlled by ones, and K and L are two m × n matrices composed by kernel functions, and the Hadamard product of K and L is used to deﬁne φ , that is φ ( v x i , v ˆ y i )
=
K ( v x i ) ◦
L ( v ˆ y i ) .
Finally, an appropriate mutual information estimator of with smaller bias is expressed as:
[Equation]
Note that, the computation of (cid:92) rSMI ( V x i , V ˆ y i ) is resource friendly, as it can be solved analytically.
Thus, the parameters in the translation neural network can be efﬁciently updated by backprogation.
3. F ULL O BJECTIVE
Following the analysis above, our minimal geometry-distortion constraint (MGC) for I2I translation using mutual information can be expressed as:
[Equation]
where N is the number of samples.
We directly maximize L mgc to guarantee more local geometric structures of images being invariant in the translation process.
By combining our MGC with the standard adversarial loss, the image geometry will be preserved while its style is changed.
As a result, one-sided unsupervised domain mapping can be targeted.
The full objective will take the form:
[Equation]
where L gan is the adversarial loss (Goodfellow et al., 2014), which introduced a discriminator
D Y , to encourage the distribution of output matches the distributions of target domain images, i.e
P G XY ( X )
≈ P Y .
The objective function as follows:
[Equation]
In Equation 7, λ mgc is a hyperparameter to weight L gan and L mgc in the training procedure.
The proposed MGC can be integrated into various I2I translation frameworks, e.g., CycleGAN (Zhu et al., 2017) and CUT(Park et al., 2020), by replacing the loss L gan with the losses in these methods. [Figure]
E XPERIMENTS
In this section, we perform experiments on three typical unsupervised geometry-invariant image translation applications: Digits Translation, Unsupervised Segmentation and Image Generation ( e.g. Cityscapes (Cordts et al., 2016) ), and Style Transfer.
To show the effectiveness of our MGC, we couple our minimal geometry-distortion constraint (MGC) with the vanilla GAN.
Further, we incorporate MGC with some popular methods such as cycle-consistency (Zhu et al., 2017), GcGAN(Fu et al., 2019), and U-GAT-IT (Kim et al., 2019) to show its compatibility with other constraints.
Then we make qualitative and quantitative comparisons with the recent published unsupervised I2I translation methods
e.g. CycleGAN, GcGAN, CoGAN (Liu & Tuzel, 2016), SimGAN (Shrivastava et al., 2017), BiGAN (Donahue et al., 2016) , DistanceGAN (Benaim & Wolf, 2017), CUT (Park et al., 2020)), the Contextual Loss (Mechrez et al., 2018), DRIT++ (Lee et al., 2019), UNIT (Liu et al., 2017), MUNIT (Huang et al., 2018), AGGAN (Tang et al., 2019), and U-GAT-IT (Kim et al., 2019). However, the current baselines have their own advantages and disadvantages: some baselines perform well on one task but perform poorly on other tasks.
For example, some style transfer methods do not perform well on unsupervised image segmentation.
As such, following the current literature, we compare our methods with SOTA methods for each application.
At last, we perform the ablation study by varying the hyper-parameter λ mgc .
In the appendix, we ﬁrstly explore the use of our MGC on the geometry-variant dataset, i.e. , cat2dog dataset (Lee et al., 2019) (the analysis is given in the A.5), and then investigate the inﬂuence of our MGC on the generation diversity A.2.2 and training stability A.2.3.
We examine all the experiments three times and report the average scores to reduce random errors.
For the implementation of the mutual information estimator presented in section 3.2, we set the hyperparameter β to 0.5 (more analysis and experiments about other values of β are given at the appendix A.2.1), and utilize nine Gaussian kernels for both input images x and translated images ˆ y .
Then we apply our MGC to all the baselines and keep other experimental details including hyper-parameters, networks in baselines the same.
Due to page limit, we provide more experimental details and qualitative results in the Appendix A.4 and A.7, respectively. [Table] GcGAN- rot 32.5 ± 2.0 95.0 ± 0. 85.9 ± 0.
40.9 ± 6. 84.6 ± 2. 96.0 ± 0. GcGAN- rot + MGC 36.5 ± 1.3 96.4 ± 0.
91.8 ± 1.
47.5 ± 1.
89.5 ± 0.
96.1 ± 0.
GcGAN - vf 33.3 ± 4.2 95.2 ± 0.
84.5 ± 1.
31.6 ± 5.
83.8 ± 3.
95.9 ± 0.
GcGAN- vf + MGC 37.0 ± 0.8 96.6 ± 0.
91.8 ± 0.
49.5 ± 4.
87.8 ± 2.
96.0 ± 0.
CycleGAN + rot + MGC 39.0 ± 0.5 96.5 ± 0.
91.8 ± 1.
50.5 ± 1.
89.8 ± 0.
96.1 ± 0.
CycleGAN + vf + MGC 44.6 ± 6.8 96.7 ± 0.
92.0 ± 0.
51.3 ± 5.
89.0 ± 0. 96.1 ± 0. D IGIT T RANSLATION Following (Fu et al., 2019; Benaim & Wolf, 2017), we examine three digit I2I translation tasks:
SVHN → MNIST, MNIST-M → MNIST and MNIST → MNIST-M 1 .
The models are trained on the training split with images size 32 × 32 , and λ mgc is set to 20.
We adopt the classiﬁcation accuracy as the evaluation metric, and design two evaluation methods: (1) we train a classiﬁer on the target dataset’s training split.
The fake images translated from the source dataset’s test images are used to compute the classiﬁcation accuracy.
This evaluation method can only measure the quality of translated images.
(2) a classiﬁer is trained on the translated images from the source dataset’s training images, and test the performance of this classiﬁer on the target dataset’s test split.
This evaluation method can measure both the quality and diversity of translation images, but it is unstable 2 .
We conduct each experiment ﬁve times to reduce the randomness of GAN-based approaches.
The scores are reported in Table 1.
Generally, by incorporating our MGC, all the baselines show promising improvements in both accuracy and stability, especially for the challenging task S
→ M.
Some qualitative results are shown in Figure 4.
More details and results are given in Appendix A.4.
and A.7.4, respectively.
4. C ITYSCAPES
Following (Fu et al., 2019; Zhu et al., 2017), we train the models using the unaligned 3975 images of
Cityscapes (Cordts et al., 2016) with 128 × 128 resolution.
We evaluate the domain mappers using
FCN scores and scene parsing metrics as previously done in (Zhu et al., 2017).
Speciﬁcally, for parsing → image, we use the pre-trained FCN-8s (Long et al., 2015) provided by pix2pix (Isola et al., 2017) to predict the segmentation label maps from translated images, then compare them against the ground truth labels using parsing metrics including pixel accuracy, class accuracy, and mean IoU. For image → parsing, the translated label maps are also compared against the ground truth.
[Table]
[Table] [Table]
As reported in Table 2, the results of all the image translation methods are improved if further constrained by our MGC.
In particular, GcGAN coupled with MGC yields a promising improvement compared with GcGAN in the parsing → image task.
4. S TYLE T RANSFER
We implement the style transfer task on anime2selﬁe (Kim et al., 2019), horse2zebra (Zhu et al., 2017), photo2portrait (Lee et al., 2018).
We choose CycleGAN, GcGAN, AGGAN, DRIT, UNIT,
MUNIT, and CUT as baselines.
All images are resized to 256 × 256 resolution
and λ mgc is set tofor all experiments.
More details are given in A.4.4.
Following the recent work (Kim et al., 2019), we use KID score (Bi´nkowski et al., 2018) as the evaluation metric.
The results are reported in the Table 3, we can see that the vanilla GAN method coupled with our MGC can achieve the comparable results with those methods with larger model size.
In addition, a simple generator based on res-blocks trained by the combination of cycle, geometry and our MGC constraint can achieve SOTA performance on almost all datasets.
As the qualitative results shown in Figure 5, after adding our MGC, the translated images retain the geometric structure of the original image, and is consistent with the style of the target image.
More results are given in the appendix A.7.3, and the light version of U-GAT-IT with our MGC can achieve better performance of full version of U-GAT-IT, at the situation of saving a half size of parameters. Then we conducted a
[Table]
[Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure]
[Table]
user study, in which 180 participants were asked to choose the best translated image given the domain names e.g. selﬁe → anime, exemplar images in the source and target domains
[Table]
and the corresponding translated images from different methods.
The results shown in Table 4 demonstrate that most users choose the outputs of our method.
4. S ENSITIVITY A NALYSIS
We study the inﬂuence of MGC by performing experiments with different λ mgc .
As shown in Tableand Figure 6, although the best λ mgc varies in each task, the performance of translation models are all improved to some extend after incorporating our MGC.
However, when λ mgc becomes too large, the improvement with our MGC is limited as the model focuses on reducing geometry distortion and ignores the style information learned from GAN.
More examples are given in the Appendix [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
A.7.5.
A practical strategy of choosing λ mgc is to ﬁnd the largest λ mgc with normal style information using binary search.
Speciﬁcally the ﬁrst value of λ mgc can be set 5, which is suitable for most style transfer datasets.
C ONCLUSION
In this paper, we propose the minimal geometry-distortion constraint (MGC) to improve geometry invariance in pixel-wise level for unsupervised image-to-image translation.
In addition, we propose an expression of mutual information called relative Squared-loss Mutual Information(rSMI) with an analytical method for estimation.
We evaluate our model quantitatively in a wide range of applications.
The experimental results demonstrate that MGC achieves high quality translation to maintain the geometry of images in original domain.
A A PPENDIX A. D ETAILS OF S OLVING α
To solve the rSMI ( V x i , V ˆ y i ) , we directly estimate the density ratio using a linear combination of kernel functions of { v x i j } Mj =1 ∈ V x i and { v ˆ y i j } Mj =1 ∈ V
ˆ y
i :
[Equation]
x [Equation] where φ ∈ R m is the kernel function, α ∈ R m is the parameter vector we need to solve, and m is the number of kernels.
α is learned so that the following squared error J ( α ) is minimized :
[Equation]
where J 0 is a constant number respect to α , and therefore can be safely ignored.
Thus, the optimization problem is given as:
[Equation]
where
[Equation]
For computational efﬁciency, we deﬁne the kernel function φ ( v x i , v ˆ y i ) as the product of K ( v x i ; k c ) ∈ R m and L ( v x i ; l c ) ∈ R m , which are kernel functions of v x i and v ˆ y i respectively:
[Equation]
where ◦ denotes the Hadamard product.
Approximating the expectations in H and h by empirical averages, and adding a quadratic regularizer α T Rα to avoid over-ﬁtting, the objective function in our optimize problem becomes:
[Equation]
where R is the positive semi-deﬁnite regularization matrix [Figure] and
[Equation]
where n is the number of samples
[Figure] 1 n is the n-dimensional vector ﬁlled by ones [Figure] and K and L are two m × n matrices composed by kernel functions.
The equation 10 is a unconstrained quadratic problem [Figure] and thus could be solved by analytically and the optimal solution of ˆ α is:
[Equation]
A. E XPERIMENTAL A NALYSIS
[Figure] A.2. β A NALYSIS
We conduct the sensitive analysis of β on the digits datasets (each experiment is repeated 3 times) and the results are shown as Figure 7 (b).
We can see the performance of translation models are all improved with varied β , and we use 0.5 for convenience.
A.2. G ENERATION D IVERSITY A NALYSIS
We conduct the generation diversity experiments on the edge2shoes dataset.
Following MUNIT (Huang et al., 2018), we calculate the average LPIPS distance between 1900 pairs of randomly generated images (sampled from 100 input images).
MUNIT with MGC has the average LPIPS of 0.120, improving the diversity of vanilla MUNIT model with 0.104 LPIPS score.
Therefore, our
MGC has no negative impact on generation diversity.
Some generation examples are given as Figure
A.2. S TABILITY A NALYSIS
We conduct the training stability analysis of our MGC on the digits datasets and the results are shown as Figure 7 (a).
We can see the training procedure is stable with MGC
A. E XPERIMENTS A.3. MAPS
The Maps dataset (Isola et al., 2017) contains 2194 aerial photo-map image pairs, with 1096 pairs for training and 1098 pairs for evaluation.
For evaluation, we employ the metrics including RMSE and pixel accuracy with threshold δ ( δ 1 = 5 and δ 2 = 10 ) suggested by GcGAN (Fu et al., 2019).
[Table]
The scores are reported in Table 6.
Our MGC can signiﬁcantly improve the accuracy to 38.6% and 61.8%, compared with the vanilla GAN, whose scores are 19.3% and 42.0% with the threshold of δand δ 2 , respectively.
Moreover, integrating our MGC constraint into CycleGAN and GcGAN can generate better translations than both individual ones.
This further demonstrates the compatibility of the proposed mutual information method.
Qualitative results are shown in Appendix A.7.1.
A. E XPERIMENTAL D ETAILS
We will public codes and experimental setting for the convenience of reproducing results in our paper.
A.4. D IGITS
All digits images are resized to 32 × 32 resolution.
Following Fu et al.
(2019), the network details of this experiment are given in Table 7. [Table]
Following all settings of the original models
[Table]
the learning rate for generator and discriminator is 0.0002, the training epochs is 40000 and the batch size is 64.
A.4. C ITYSCAPES
All images are resized to 128 × 128 resolution.
Following Zhu et al.
Fu et al.
(2019), the network details of this experiment are given in Table 8.
[Table]
Following all settings of the original models [Table] the learning rate for all generators and discriminators is 0.0002, the batch size is 1 and the training epochs for CUT is 400 and other models is 200.
A.4. M APS
All images are resized to 256 × 256 resolution.
Following Zhu et al.
Fu et al.
(2019), the network details is similar to the details of Cityscape, but the generator contains 9 res-blocks for images with 256 × 256 resolution.
Following all settings of the original models, the learning rate for all generators and discriminators is 0.0002, the batch size is 1 and the training epochs for CUT isand other models is 200.
A.4. S TYLE T RANSFER
All settings are same with Maps A.4.3.
The details of datasets as follows:
selﬁe2anime
This dataset is from U-GAT-IT Kim et al.
(2019), which contains 3400 training images and 100 images for test.
horse2zebra
This dataset is from CycleGAN Zhu et al.
(2017), whose training sets contains 1, horse images and 1,334 zebra images.
The test set consists of 120 horse images and 140 zebra images.
portrait2photo
This dataset is from DRIT Lee et al.
(2019), whose training sets contains 6, photo images and 1,811 portrait images.
The test set consists of 751 photo images and 400 portrait images.
Following all settings of the original models, the learning rate for all generators and discriminators is 0.0002 and the training epochs for CUT is 400 and other models is 200.
A. A NALYSIS ON THE C AT 2D OG D ATASET
To analyze the performance of our MGC on geometry-variant datasets, we incorporate our MGC constraint into CycleGAN model and train it on the cat → dog dataset.
The results are shown as
Figure 10 , we can see that the trained translation model can successfully translate dog images at the top row to cat images and preserve the basic image content ( i.e. locations of eyes, mouth, directions of faces), even if there are some changes of geometric structure.
However, as images at the bottom row show, the translation model fails to translate the dog images to cat images in a meaningful way, as the mouth of dogs block the background but the mouth of cats do not, and so the translation model need to "imagine" some background area that be blocked, which needs us to propose more constraints.
[Figure]
A. F AILURE C ASES A NALYSIS
Our MGC is a general geometry preservation constraint for I2I translation, aiming to reduce the randomness of color transformation in the translation. However, the style information can only be learned by GAN or other methods (e.g. U-GAT-IT). Limited by the absence of supervision signal, these methods sometimes recognize the semantic information wrongly, e.g. the background should be preserved in horse2zebra, but translation models sometimes mis-recognize some background to a horse in horse2zebra, and thus produce some zebra texture in the background in the translated image (best viewed at zoom level 200%). After coupling with our MGC, the background similar to the mis-recognized background is also likely be mistranslated (e.g. more background is mistranslated to texture of zebra).
Although there are a small number of such cases occur in the translation process, the geometric details (e.g. sea or beach or sky) that are not mis-recognized can be better preserved with our MGC. [Table] [Table] [Table]
A.7. C ITYSCAPES
[Figure]
[Table]
[Figure]
[Table]
A.7. S TYLE T RANSFER
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Figure] [Figure]
[Figure]
An example of SVHN to MNIST translation when λ mgc is set to 25 is shown as Figure 13.
The images are almost translated without any changes in geometry structures.
However, the overlarge λ mgc causes the translation model neglect the style information from adversarial loss, resulting in some images with opposite color.
This phenomenon indicates that our MGC has good performance on the preservation of geometry structure but should be appropriate with style information.