Unsupervised Image-to-Image Translation with Density Changing Regularization
Anonymous Author(s)
AffiliationAddress email
Abstract
Unsupervised image-to-image translation aims to translate an input image to another domain such that the output image looks like an image from another domainwhile important semantic information are preserved.
Inferring the optimal mappingwith unpaired data is impossible without making any assumptions.
In this paper,we make a density changing assumption where image patches of high probabilitydensity should be mapped to patches of high probability density in another domain.
Then we propose an efficient way to enforce this assumption: we train the flowsas density estimators and penalize the variance of density changes.
Despite itssimplicity, our method achieves the best performance on benchmark datasets andneeds only 56 − 86% of training time of the existing state-of-the-art method
Introduction
Unsupervised image-to-image translation aims to translate an input image to another domain suchthat the output image looks like an image from another domain while important semantic informationare preserved.
For example, in the task selfie → anime, we need to translate the selfie photo into animestyle while the identity of input human is still preserved.
In addition, many research problems can bereformulated as image translation tasks, such as domain adaptation [18, 34], medical image analysis[2] and image super-resolution [48].
In unsupervised image translation setting, we are given two collections of samples without pairinginformation and we need to learn a proper mapping from one domain to another.
Unfortunately,given two marginal distributions, there can be infinite number of joint distributions that can derivethe same marginals [30].
Therefore, assumptions are needed to address this ill-posed problems.
Onepopular assumption is the cycle consistency [53, 26, 46], which assumes that the proper mapping isone-to-one.
Cycle consistency has been shown to achieve impressive visual performance in manytasks [53, 25].
But sometimes the one-to-one assumption can be over restrictive, especially for sometasks that one domain has more information than another one [39].
As an alternative, contrastivelearning based image translation is gaining wide attention [39].
CUT [39] employs infoNCE loss[42] to maximize the mutual information between two corresponding patches in the input and outputimages.
Recent image translation methods are mostly trying to improve CUT, such as negative samplemining [43, 24], informative sample mining [20, 49] and dual embedding learning [16].
In this paper, we propose a different way to find the optimal mapping by looking at the neighboringinformation.
DistanceGAN [4] proposes to maintain the pairwise distance between images after themapping.
But the pairwise distance is usually computed within mini-batches and the neighboringinformation can be inaccurate, which may leads to unsatisfactory performance as reported in [39].
Wepropose to consider the neighboring information from the density perspective: the probability densityof an image or patch (a smaller piece of the image) is a good representation of its neighborhood.
Ifthe probability density of an image is high, it means that many samples are likely to be drawn from its neighborhood.
In light of this analysis, we propose our density changing regularization: patcheswith high (low) density should be mapped to patches with high (low) density in another domain .
Asillustrated in Figure 1, the gray area in input image (blue box) is of higher density than the greenarea (orange box), then our regularization encourages that the gray area is mapped to area with highdensity in another domain.
In this example, the patch of building is of higher density than the patchof trees.
Therefore, our regularization encourages the gray area to be mapped to the building and thegreen area to be mapped to the trees.
As we can observe from the ground truth, it is exactly whatthe ground truth photo does.
The regularization is implemented in a very simple way: we just traintwo density estimators and penalize the variance of the density changes of patches.
We apply ourmethod to different image translation tasks and the superior performance across tasks
demonstratethe effectiveness of our method.
In addition, our method needs only around 56-86% training time ofstate-of-the-art (SOTA) methods.
Related Work
Image Translation For unsupervised image-to-image translation, we are given only two collectionsof images and we need to learn a proper mapping from one domain to another.
However, the numberof possible mappings can be infinite and we need to introcue additional assumptions to reduce it.
Cycle consistency [53, 46, 26] is one of the most popular assumptions.
It enforces the mappingfunction to be one-to-one and has shown success in many tasks [25, 1, 7, 6].
However, it may berestrictive when the optimal mapping is not one-to-one, e.g., segmentation map to a real photo.
Toaddress this issue, CUT [39] proposes a contrastive learning based method.
It maximizes the mutualinformation between two corresponding patches in the input and output image with the infoNCEloss [37].
Unlike CycleGAN [53], CUT doesn’t need two mapping function and allows one-sidedtraining.
However, as many recent method revealed, the effectiveness of the method can be hamperedby the negative samples used.
Randomly selected negative samples might not be useful.
Therefore,recent methods are mostly focusing on mining more meaningful negative samples or positive samplesfor better results [24, 43, 49, 20].
Unlike the mutual information based methods, our method focuson the neighboring information of patches and no negative sample are needed.
There are also somemethods arguably based on the neighborhood information.
DistanceGAN [4] proposes to maintain the pairwise distance between images in the source domain and the translated domain.
HarmonicGAN[50] proposes a smoothed distance objective inspired by [54].
F-LSeSIM [52] proposes to encouragethe pairwise distance matrix of input and translated patch representations to be consistent.
However,the pairwise distance loss may be inaccurate due to the insufficient sample and the training objectivemay also be dominated by the very distant pairs.
In contrast, our method utilizes the probabilitydensity, which represents the neighborhood information of the whole dataset and we also avoid thepossibly bad influence by the very distant pairs.
There are also some powerful alternatives, such asthe spatial transformation consistency [12, 45] and shared latent space assumption [30, 22, 29].
Flow+GAN FLOW-GAN [15] uses the normalizing flow as the generator.
Then the generator canbe trained by maximum likelihood and adversarial objectives hybridly.
Based on FLOW-GAN,
Alignflow[14] first translates the source image into a latent variable with flow and generate itscorresponding image in the target domain by feeding the latent variable into another flow.
Since flowsare designed to be invertible, the mapping function is exactly invertible while CycleGAN [53] onlyenforces that property approximately.
iFlowGAN [9] proposes to regularize the lipschitz norm ofeach layer and thus the mapping function is invertible and the inverse can be obtained by fixed-pointtheorem.
Unfortunately, the flow model can take very long time to train and has inferior performancecompared to GAN model in high-resolution image modeling.
Our model doesn’t need totrain the flow as image modeling but we utilize its strong density estimation ability to regularize the
GAN training.
Density Estimation Estimating the density P ( x ) from a set of samples { x i } is an important problemin machine learning and computer vision.
There have been many successful works that estimate thedensity with neural networks [38, 41, 28, 11, 21, 23, 31, 13, 44, 36, 19].
In this paper, we considerthe autoregressive normalizing flow as our density estimator.
The flow model f maps the data into
Gaussian noise and the density of data x can be computed as p ( x ) = p ( z ) | det J f ( x ) | , where
z = f ( x ) ,p ( z ) is the normal distribution density and | det J f ( x ) | is the absolute value of determinant of the
Jacobian matrix.
With careful design of the architecture of the function f , the determinant can alsobe computed efficiently.
Proposed Method
Given images { x i } ∈ X and images { y j } ∈ Y , our goal is learn a mapping G : X → Y suchthat G ( x ) looks like images in the domain Y while preserving necessary semantic information.
For example, in translating a segmentation map to real-world photo, we need to ensure that generatedphoto looks realistic while the contents (e.g., building, trees, roads) in the generated photo follow thesegmentation map.
3.1 Baseline GAN model
To achieve the first goal that G ( x ) looks like an image in another domain Y , we employ an adversarialloss with the domain discriminator D to match the distribution between P G ( X ) and P Y , L gan = E x ∼X [log( [Equation]
D
[Equation]
G
[Equation]
x )))]
[Equation]
∼Y [log
[Equation]
D [Equation] y ))] . We also additionally introduce an identity mapping loss to regularize the mapping function G , whichis widely adopted in existing image translation methods [53, 39].
[Equation]
[Equation]
By combing these two losses, we obtain an baseline GAN model for image translation.
Unfortunately,there are can still be infinite number of mappings that satisfy these objectives and we need furtherconstraints to reduce the number of possible mappings in the space [53].
3.2 Proposed Density Constraint
In this paper, we consider the neighboring information from the probability density perspective.
Wefirst revisit the definition of density function: Let X be a continuous real-valued random variable.
Adensity function for X is a real-valued function f which satisfies
[Equation]
for all a, b ∈ R .
Given a value x of X , we have P ( x − δ ≤ X ≤ x + δ ) =
(cid:82) x
+ δ x − δ f ( x ) dx
≈
δf ( x ) .
For a predefinedδ , if the density of x is high, then P ( x − δ ≤ X ≤ x + δ ) is high.
It means that there are will bemany samples that fall within the range [ x − δ, x + δ ] .
Therefore, there will be many neighbors ofx in the sample space.
It follows that the density f ( x ) can be viewed as a measure of how manyneighbors (within δ distance) x will have.
In light of this observation, we propose our density changing constraint for unsupervised image-toimage translation: if the density f ( x ) of a image x is high in the domain X , then the translation G ( x )should also have high density P ( G ( x )) in domain Y .
The density function provides us an elegantway to utilize the neighboring information without computing the time-consuming and possiblyinaccurate pairwise distances.
Given two density estimators f X , f Y for two domains, we have ourconstraint defined as
[Equation]
where V ( . ) is the variance function.
It means that we would like the density changes for all imagesto be close such that image with high density is still mapped to high density while image with lowdensity still mapped to low density.
In the following sections, we provide an efficient way to get thedensity estimators f X , f Y and the density changing objective L density .
Another perspective of density changing constraint .
From the definition, we can see that weare enforcing constraint on the probability density ratios.
If the density estimators are accurate,i.e., f X ( x ) = f ( x ) , f Y ( G ( x ))
=
f ( G ( x )) and the mapping function G is invertible, then we arehaving | det ( J G ( x ) ) | = f ( x ) f ( G ( x )) according to the change of variable formula, where | det ( J G ( x ) ) | is theabsolute value of determinant of the Jacobian matrix of the mapping function G ( x ) . This term canbe viewed as a measure of volume changing caused by the function G in the space.
Therefore, byminimizing the L density when G is invertible, we are enforcing that mapping function G have similarvolume changing for all samples.
3.3 Density Changing Regularized Unpaired Image Translation
To compute L density , we need density estimators f X and f Y .
There are many successful autoregressiveflows for density estimation [21, 10].
However, density estimation for images is still an challengingtask due to the high-dimensionality of images.
Inspired by the recent patch-level image translationmethod [39], we propose to compute the densities in the patch level rather than image level.
We firstextract the feature maps at layer l of the generator G as m l ( x ) =
G l ( x ) ∈ R c × h × w , where c is thenumber of channels of the feature map and h, w represents the height and width of the maps.
Thenwe reshape m l ( x ) into patch representations P l ( x ) ∈ R hw × c .
For each representation of size c , itcorresponds to a patch in the input image.
And we have hw patch representations.
We denote thepatch representations of input image x ∈ X as P l ( x ) = { a li } , input image y ∈ Y as P l ( y ) =
{ c li } ,generated image G ( x ) as P l ( G ( x ))
= { b li } .
We sometimes use a l to denote the random variablewhose values are a li and similar for b l and c l .
For each layer l , we employ two density estimators f l X and f l Y , which are parameterized as autoregressive flows [10].
Given patch representations of real images { a li } and { b li } , we train the densityestimators with maximum likelihood estimation as
[Equation]
where the outputs of the flows f X and f Y are log-probabilities and we need to minimize above lossto maximize the log-likelihood.
Given the density estimators, we can compute our density regularization easily as
[Equation]
The outputs of density estimators are log-likelihoods and therefore, f l X ( a l ) − f l Y ( b l ) is equivalent tothe probability ratios in the log-scale since log pq = log p − log q .
Please note that a l and c l representsthe corresponding patches in the input image x and output image G ( x ) .
By minimizing L patch-density ,we are encouraging the transformation from a l to b l should satisfy our assumption, i.e., patches withhigh density should be mapped to patches with high density.
Full Objective Our full objective for the generator consists of three terms
[Equation]
where λ idt and λ density are hyper-parameters that balance different losses.
For the discriminator, we train it to maximize the L gan to distinguish the real from the generatedimages.
[Equation]
For the density estimators, we train them with L nll as
[Equation]
For brevity, we denote our method as DEnsity Changing rEgularized uNpaired image Translation(DECENT).
Implementation
Density Estimator .
In our initial experiments, we consider three auto-regressive neural flows fordensity estimator: MAF [38], BNAF [10] and NSF [11].
But we observe that MAF and NSFsometimes reports NAN error, which may be caused by the exponential scaling function.
BNAFfurther adopts the weight normalization technique to stablize the training.
Therefore, we stick to
BNAF in our experiments.
We use the recommended hyper-parameters by BNAF: we set the hiddendim as 10, polyak β 1 =0.9, β 2 = 0.999. To reduce the computation cost, we set the number of blocks in BNAF to 1. Since there are 5 layers used to extract patch representations, we employ 5 BNAF for each domainin total. The number of parameters of the 5 BNAFs is 4.280 M. As a reference, the number ofparameters of the generator is 11.378M.
Generator and Discriminator .
We follow the networks used in CUT [39].
For the generator, weadopt the 9-block ResNet as the backbone network.
For the discriminator, we adopt the 3-layer
PatchGAN discriminator.
The learning rate is 0.0002 with β 1 = 0 .
5 , β 2 = 0 .
The exceptionis the cat → dog task, which requires large shape changes.
Following NEGCUT[43], we replace thepatchGAN discriminator with a 6-layer discriminator, which captures more global information.
Hyper-parameters .
We mostly follow the hyper-parameters in CUT [39].
We set L = 5 following CUT [39].
It means that we use five layers (0,4,8,12,16th) of the generator to extract patch representations.
The layers correspond to receptive fields of sizes 1 × 1, 9 × 9, 15 × 15, 35 × 35, and 99 × 99.
We also use 256 patches in each layer instead of all patches to save computation time and memory.
The difference is λ idt . CUT [39] uses contrastive learning to achieve this identity while we use L 1reconstruction loss. Therefore, we set it to λ idt = 10 following [53, 25]. The only hyper-parameterwe introduce is λ density .
We initially set it to 1 but found that it is over-restrictive and causes modelcollapse.
We next tried two values { 0 . 01 , 0 . 1 } , and discovered both work pretty well, with 0.01performing slightly better.
We set λ density = 0 .
01 across all tasks.
Training In each iteration, we first update the density estimators by minimizing L mle .
Then weupdate the discriminator by L dis .
When optimizing the generator, to compute L patch-density , we usethe polyak averaged version of BNAF to estimate the density for patches.
For L gan , we replace thevanilla GAN loss with the LSGAN [33] objective following CUT [39].
For all dataset, we resizeimages to 256 × 256.
We keep the learning rate for the first half of the training and linearly decay it to0 in the last half of training.
Experiments5.1 Evaluation
We follow the evaluation protocols in [53, 39, 43, 20, 49] and run all tasks once.
We report results ofmore runs in the supplementary material.
Datasets We follow the evaluation protocols in CUT [39] by running experiments on three benchmarkdatasets: label → city, cat → dog and horse → zebra.
We further run experiment on selfie → anime datasetto fully verify the effectiveness of our method.
The label → city dataset [8] consists of 2975 trainingimages in each domain and 500 pairs for testing.
The cat → dog [7] dataset contains around 5000training images in each domain and 500 testing images.
The horse → zebra dataset [53] containsaround 1200 training images in each domain and around 100 testing images.
The selfie → animedataset [25] contains 3400 training images in each domain and 100 testing images.
We run label → cityand horse → zebra for 400 epochs, cat → dog and selfie → anime for 200 epochs.
Metrics For the label → city task, we follow [ 39 ] and evaluate the generated photos by a pre-trainedsegmentation model DRN [47].
The DRN model translates the generated photos into segmentationlabels and then we can compare the output labels with the input labels.
We compute mean averageprecision (mAP), pixel-wise accuracy (pAcc) and average class accuracy (cAcc).
To evaluate the visual quality of generated images, we also adopt the widely-used Frechet Inception
Distance (FID) [17] and Kernel Inception Distance (KID) [5] scores.
FID and KID both measure thedistribution divergence between the generated images and the real images.
Baselines For the first three tasks, we compare our method with different assumptions: cycleconsistency-CycleGAN [53], pairwise distance preservation – DistanceGAN and SelfDistance [4]and mutual information maximization – CUT[39], DCLGAN [16], FSeSIM [52], NEGCUT [43],
MoNCE [49] and QS-Attn [20].
Please note that the MoNCE [49] and QS-Attn [20] are released lessthan two months before this submission and they serve as strong baselines.
For selfie → anime, we compare with CycleGAN [53], U-GAT-IT [25], CUT[39], CouncilGAN[35], ACL-GAN [51] and SpatchGAN [40].
For all tasks, we also run the Base-GAN model, i.e.,λ density = 0 , to fully examine the effectiveness of our proposed regularization.
[Table]
[Table]
[Table]
[Table]
Comparison against Baselines
We present the quantitative results in Table.
1 and 2.
Among the four tasks, our method achievesthe best performance on the label → city, cat → dog tasks and selfie → anime tasks.
In particular, ourmethod is the first one to improve mAP above 30 on the label → city task.
Please note that all recentstate-of-the-art methods are improved versions of CUT [39].
By contrast [Table] we propose an effectivenessregularization which is orthogonal to the contrastive learning [39] and cycle consistency losses[53].
Our method brings substantial improvements over the Base-GAN model.
For the label → citytask, we outperform it by 41.7% on mAP, 35.4% on pAcc and 36.5% on cAcc.
The significantimprovements demonstrates that our proposed density regularization is able to preserve importantsemantic information in inputs.
On the cat → dog task, our method improves the FID from 140.1 to55.2.
On the selfie → anime task, our method improves the FID from 89.5 to 80.7 and KID from3.09 to 1.42.
The encouraging results suggest that our regularization is also helpful in distributionmatching.
We also present the generated samples in Figure.
3 and more samples are provided in the supplementary material.
For the label → city task, we can observe that the SOTA method– QS-Attn [20] stillfaces some label flipping issue, i.e., mapping the gray area to the trees.
By contrast, our method isable to map the gray area into the building accurately.
For the cat → dog task, we can observe that the
Base-GAN model suffers the well-known mode collapse issue while our method is able to avoid it byour proposed regularization.
For the selfie → anime, our method learns to preserve the hair color ofthe first input person while keeping the headphone cable in the output image.
Unlike CouncilGAN[35] and SpatchGAN [40], our method doesn’t apply additional data augmentations, such as randombrightness and hue offset.
5.3 Analysis
Sensitivity of λ density .
Although our method only introduces one loss, it is still important to investigatethe sensitivity of the introduced hyper-parameter λ density .
We run our method on label → city task since
Input
QS [Figure] Attn [20]
DECENT
Truth
[Figure]
the testing images are paired.
The results are shown in Table.
We can observe that our methodis robust to the scale of λ .
When λ = 0 .
01 or 0 . 1 , we can achieve state-of-the-art results.
Whenλ = 0 .
0001 , it still improves the Base-GAN model from 21.86 to 28.15 on mAP metric.
When weincrease it to 0.5, which is quite far from the optimal value, we can still observe large improvementsover the Base-GAN model.
Pairwise relationship vs Density We can observe that the performance of DistanceGAN and
SelfDistance [4] are unsatisfactory in Table.
As we have argued in section 2, it may be caused byinsufficient samples.
Therefore, we build a stronger version of DistanceGAN – Patchdist.
Patchdist istrained to preserve the pairwise distance between all patch representations now.
Since many patchesare available in each iteration, the pairwise relations should be more accurate.
As shown in Table.
PatchDist can also bring improvements over Base-GAN model now (DistanceGAN and SelfDistanceare worse).
The improvements supports our argument that the neighboring information is very usefulin image translation tasks.
Our method still outperforms PatchDist by a large margin, which impliesthat the densities values are more informative than the pairwise relation in image translation tasks.
Training Time
We run all methods on NVIDIA Tesla V100-SXM2 GPU and report their trainingspeed in table 2.
One may notice that DistanceGAN [4], SelfDistance [4] and FSeSim [52] are fasterthan the Base-GAN model.
The main reason is that they don’t apply the identity regularization.
When compared to the three SOTA methods (NEGCUT [43], MoNCE [49] and QS-Attn [20]), our method achieves the best performance with the highest training speed.
NEGCUT needs togenerate negative samples with additional networks, MoNCE needs to compute the entropy of allpatch representations and MoNCE needs to address an optimal transport problem every iteration.
In contrast, we only introduce the density estimators and avoids the heavy global informationcomputation.
As a consequence, our method only needs approximately 56%, 67% and 86% trainingtime of NEGCUT [43], MoNCE [49] and QS-Attn [20], respectively.
Learned densities We visualize the learned densities in Figure.4.
The densities increase as thecolor goes from blue to red.
Firstly, we observe that our learned densities are consistent with humanjudgements and reveal the neighboring information accurately.
For example, human can easily tellthat gray patches should have more neighbors than green patches in the first row.
The densities ofgray patches shown in the second row are higher than the green patches.
Then we examine whether our method effectively enforces our assumption and whether it bringsimprovements.
On the label → city task, we observe that the mapping of Base-GAN violates ourassumption, i.e., Base-GAN maps the green patches (low density) to the building (high density)
whilegray patches (high density) to the trees (low density).
As a consequence, it suffers the label-flippingissue when compared with the ground truth photo.
In contrast, our method is able to find the correctmapping through our density regularization.
Limitation and Discussion
Although our method achieved state-of-the-art performance across many tasks, our method stillsuffers from some limitations: (1) the violation of the assumption; One may notice that the BaseGAN model achieves the second best FID on the horse → zebra task as shown in Table.
Althoughour method is slightly better than other SOTA methods, it still brings some performance degradation(40.1 → 41.3).
This may be caused by the unmatched dataset statistics since it is reported that horsetakes 18% pixel of the image and zebra takes 37% pixels in the dataset [39].
Therefore, patchesof high density in the horse domain may not still be of high density.
A possible solution can beadopting the attention module to only enforce our density regularization on the important patches.(2) patch correspondence.
Using representation at patch level provides more samples than imagelevel.
But our method also faces the same limitation as patch-based methods [39, 20, 49]: the patchesmay not be well corresponded in some tasks.
For example, when the first domain is the front facewhile the second domain is the face profiles, we need to learn the rotation mapping between twodomains, which may cause patch-based methods to fail.
A possible solution would be consideringlarger patches, i.e., increase the number of layer used to extract patch representations.
Large patchesought to contain more global information and may address this issue.
In summary, we propose a newmethod to address the unsupervised image translation problem.
We test it on multiple tasks and theyachieve best performance across tasks.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] , weprovide it in the supplemenary material.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[N/A] , we follow the evaluation protocols in existing methods
and run once.
But we provide results of more runs in the supp.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] , we provide it in the supplementary.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] , we provide
discussions to this section in the supplementary.
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] we
present more generated samples for better comparison
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]