Unpaired Image-to-Image Translation with Density Changing Regularization
Anonymous Author(s)
AffiliationAddress email
Abstract
Unpaired image-to-image translation aims to translate an input image to anotherdomain such that the output image looks like an image from another domain whileimportant semantic information are preserved.
Inferring the optimal mapping withunpaired data is impossible without making any assumptions.
In this paper, wemake a density changing assumption where image patches of high probabilitydensity should be mapped to patches of high probability density in another domain.
Then we propose an efficient way to enforce this assumption: we train the flowsas density estimators and penalize the variance of density changes.
Despite itssimplicity, our method achieves the best performance on benchmark datasets andneeds only 56 − 86% of training time of the existing state-of-the-art method
Introduction
Unpaired image-to-image translation aims to translate an input image to another domain such that theoutput image looks like an image from another domain while important semantic information arepreserved.
For example, in the task selfie → anime, we need to translate the selfie photo into animestyle while the identity of input human is still preserved.
In addition, many research problems can bereformulated as image translation tasks, such as domain adaptation [18, 34], medical image analysis[2] and image super-resolution [49].
In unpaired image translation setting, we are given two collections of samples without pairinginformation and we need to learn a proper mapping from one domain to another.
Unfortunately,given two marginal distributions, there can be infinite number of joint distributions that can derivethe same marginals [30].
Therefore, assumptions are needed to address this ill-posed problems.
Onepopular assumption is the cycle consistency [54, 26, 47], which assumes that the proper mapping isone-to-one.
Cycle consistency has been shown to achieve impressive visual performance in manytasks [54, 25].
But sometimes the one-to-one assumption can be over restrictive, especially for sometasks that one domain has more information than another one [39].
As an alternative, contrastivelearning based image translation is gaining wide attention [39].
CUT [39] employs infoNCE loss[43] to maximize the mutual information between two corresponding patches in the input and outputimages.
Recent image translation methods are mostly trying to improve CUT, such as negative samplemining [44, 24], informative sample mining [20, 50] and dual embedding learning [16].
In this paper, we propose a different way to find the optimal mapping by looking at the neighboringinformation.
DistanceGAN [5] proposes to maintain the pairwise distance between images after themapping.
But the pairwise distance is usually computed within mini-batches and the neighboringinformation can be inaccurate, which may leads to unsatisfactory performance as reported in [39].
Wepropose to consider the neighboring information from the density perspective: the probability densityof an image or patch (a smaller piece of the image) is a good representation of its neighborhood.
Ifthe probability density of an image is high, it means that many samples are likely to be drawn from its neighborhood.
In light of this analysis, we propose our density changing regularization: patcheswith high (low) density should be mapped to patches with high (low) density in another domain .
Asillustrated in Figure 1, the gray area in input image (blue box) is of higher density than the greenarea (orange box), then our regularization encourages that the gray area is mapped to area with highdensity in another domain.
In this example, the patch of building is of higher density than the patchof trees.
Therefore, our regularization encourages the gray area to be mapped to the building and thegreen area to be mapped to the trees.
As we can observe from the ground truth, it is exactly whatthe ground truth photo does.
The regularization is implemented in a very simple way: we just traintwo density estimators and penalize the variance of the density changes of patches.
We apply ourmethod to different image translation tasks and the superior performance across tasks
demonstratethe effectiveness of our method.
In addition, our method needs only around 56-86% training timeof state-of-the-art (SOTA) methods.
From the successful experimental results, we suggest that ourmethod can be safely applied when preserving the neighboring information is needed, e.g., label → citytask.
However, the method may be less effective when preserving the neighboring information is notnecessary.
For example, for the horse → zebra task, the densities of horse patches are lower than thedensities of zebra patches.
So enforcing the regularization may be over strong on this specific task.
Inthe future, we may consider applying attention on important patches only to address this issue.
Related Work
Image Translation For unpaired image-to-image translation, we are given only two collections ofimages and we need to learn a proper mapping from one domain to another.
However, the numberof possible mappings can be infinite and we need to introcue additional assumptions to reduce it.
Cycle consistency [54, 47, 26] is one of the most popular assumptions.
It enforces the mappingfunction to be one-to-one and has shown success in many tasks [25, 1, 8, 7, 3, 41].
However, it maybe restrictive when the optimal mapping is not one-to-one, e.g., segmentation map to a real photo.
Toaddress this issue, CUT [39] proposes a contrastive learning based method.
It maximizes the mutualinformation between two corresponding patches in the input and output image with the infoNCEloss [37].
Unlike CycleGAN [54], CUT doesn’t need two mapping function and allows one-sidedtraining.
However, as many recent method revealed, the effectiveness of the method can be hamperedby the negative samples used.
Randomly selected negative samples might not be useful.
Therefore,recent methods are mostly focusing on mining more meaningful negative samples or positive samplesfor better results [24, 44, 50, 20].
Unlike the mutual information based methods, our method focuson the neighboring information of patches and no negative sample are needed.
There are also somemethods arguably based on the neighborhood information.
DistanceGAN [5] proposes to maintain the pairwise distance between images in the source domain and the translated domain.
HarmonicGAN[51] proposes a smoothed distance objective inspired by [55].
F-LSeSIM [53] proposes to encouragethe pairwise distance matrix of input and translated patch representations to be consistent.
However,the pairwise distance loss may be inaccurate due to the insufficient sample and the training objectivemay also be dominated by the very distant pairs.
In contrast, our method utilizes the probabilitydensity, which represents the neighborhood information of the whole dataset and we also avoid thepossibly bad influence by the very distant pairs.
There are also some powerful alternatives, such asthe spatial transformation consistency [12, 46] and shared latent space assumption [30, 22, 29].
Flow+GAN FLOW-GAN [15] uses the normalizing flow as the generator.
Then the generator canbe trained by maximum likelihood and adversarial objectives hybridly.
Based on FLOW-GAN,
Alignflow[14] first translates the source image into a latent variable with flow and generate itscorresponding image in the target domain by feeding the latent variable into another flow.
Since flowsare designed to be invertible, the mapping function is exactly invertible while CycleGAN [54] onlyenforces that property approximately.
iFlowGAN [9] proposes to regularize the lipschitz norm ofeach layer and thus the mapping function is invertible and the inverse can be obtained by fixed-pointtheorem.
Unfortunately, the flow model can take very long time to train and has inferior performancecompared to GAN model in high-resolution image modeling.
Our model doesn’t need totrain the flow as image modeling but we utilize its strong density estimation ability to regularize the
GAN training.
Density Estimation Estimating the density P ( x ) from a set of samples { x i } is an important problemin machine learning and computer vision.
There have been many successful works that estimate thedensity with neural networks [38, 42, 28, 11, 21, 23, 31, 13, 45, 36, 19].
In this paper, we considerthe autoregressive normalizing flow as our density estimator.
The flow model f maps the data into
Gaussian noise and the density of data x can be computed as p ( x ) = p ( z ) | det J f ( x ) | , where
z = f ( x ) ,p ( z ) is the normal distribution density and | det J f ( x ) | is the absolute value of determinant of the
Jacobian matrix.
With careful design of the architecture of the function f , the determinant can alsobe computed efficiently.
Proposed Method
Given images { x i } ∈ X and images { y j } ∈ Y , our goal is learn a mapping G : X → Y suchthat G ( x ) looks like images in the domain Y while preserving necessary semantic information.
Forexample, in translating a segmentation map to real-world photo, we need to ensure that generatedphoto looks realistic while the contents (e.g., building, trees, roads) in the generated photo follow thesegmentation map.
3.1 Baseline GAN model
To achieve the first goal that G ( x ) looks like an image in another domain Y , we employ an adversarialloss with the domain discriminator D to match the distribution between P G ( X ) and P Y ,
[Equation]
We also additionally introduce an identity mapping loss to regularize the mapping function G , whichis widely adopted in existing image translation methods [54, 39].
[Equation]
By combing these two losses, we obtain an baseline GAN model for image translation.
Unfortunately,there can still be infinite number of mappings that satisfy these objectives and we need furtherconstraints to reduce the number of possible mappings in the space [54].
3.2 Proposed Density Constraint
In this paper, we consider the neighboring information from the probability density perspective.
Wefirst revisit the definition of density function: Let X be a continuous real-valued random variable.
Adensity function for X is a real-valued function f which satisfies
[Equation]
for all a, b ∈ R .
Given a value x of X , we have P ( x − δ ≤ X ≤ x + δ ) =
(cid:82) x
+ δ x − δ f ( x ) dx
≈
δf ( x ) .
For a predefinedδ , if the density of x is high, then P ( x − δ ≤ X ≤ x + δ ) is high.
It means that there are will bemany samples that fall within the range [ x − δ, x + δ ] .
Therefore, there will be many neighbors ofx in the sample space.
It follows that the density f ( x ) can be viewed as a measure of how manyneighbors (within δ distance) x will have.
In light of this observation, we propose our density changing constraint for unpaired image-to-imagetranslation: if the density f ( x ) of a image x is high in the domain X , then the translation G ( x ) shouldalso have high density P ( G ( x )) in domain Y .
The density function provides us an elegant way toutilize the neighboring information without computing the time-consuming and possibly inaccuratepairwise distances.
Given two density estimators f X , f Y for two domains, we have our constraintdefined as
[Equation]
where V ( . ) is the variance function.
Variance is a smooth function that depicts the variance of arandom quantity as a function of its mean.
If density changes of some patches are too high or toolow, the variance can be large.
It means that we would like the density changes for all images to beclose such that image with high density is still mapped to high density while image with low densitystill mapped to low density. In the following sections, we provide an efficient way to get the densityestimators f X , f Y and the density changing objective L density .
3.3 Density Changing Regularized Unpaired Image Translation
To compute L density , we need density estimators f X and f Y .
There are many successful autoregressiveflows for density estimation [21, 10].
However, density estimation for images is still an challengingtask due to the high-dimensionality of images.
Inspired by the recent patch-level image translationmethod [39], we propose to compute the densities in the patch level rather than image level.
We firstextract the feature maps at layer l of the generator G as m l ( x ) =
G l ( x ) ∈ R c × h × w , where c is thenumber of channels of the feature map and h, w represents the height and width of the maps.
Thenwe reshape m l ( x ) into patch representations P l ( x ) ∈ R hw × c .
For each representation of size c , itcorresponds to a patch in the input image.
And we have hw patch representations.
We denote thepatch representations of input image x ∈ X as P l ( x ) = { a li } , input image y ∈ Y as P l ( y ) =
{ c li } image G ( x ) P l ( G x ))
{ b li }
For each layer l , we employ two density estimators f l X and f l Y , which are parameterized as autoregressive flows [10].
Given patch representations of real images { a li } and { c li } , we train the densityestimators with maximum likelihood estimation as
[Equation]
where the outputs of the flows f X and f Y are log-probabilities and we need to minimize above lossto maximize the log-likelihood.
Given the density estimators, we can compute our density regularization easily as
[Equation]
The outputs of density estimators are log-likelihoods and therefore, f l X ( a l ) − f l Y ( b l ) is equivalent tothe probability ratios in the log-scale since log pq = log p − log q .
Please note that a l and b l representsthe corresponding patches in the input image x and output image G ( x ) .
By minimizing L patch-density ,we are encouraging the transformation from a l to b l should satisfy our assumption, i.e., patches withhigh density should be mapped to patches with high density.
Full Objective Our full objective for the generator consists of three terms
[Equation]
where λ idt and λ density are hyper-parameters that balance different losses.
For the discriminator, we train it to maximize the L gan to distinguish the real from the generatedimages:
[Equation]
For the density estimators, we train them with L nll as
[Equation]
[Equation]
X
[Equation]
} Ll [Equation] [Equation] For brevity, we denote our method as DEnsity Changing rEgularized uNpaired image Translation(DECENT).
Implementation
Density Estimator .
In our initial experiments, we consider three auto-regressive neural flows fordensity estimator: MAF [38], BNAF [10] and NSF [11].
But we observe that MAF and NSFsometimes reports NAN error, which may be caused by the exponential scaling function.
BNAFfurther adopts the weight normalization technique to stablize the training.
Therefore, we stick to
BNAF in our experiments.
More details are provided in the supplementary.
Generator and Discriminator .
We follow the networks used in CUT [39].
For the generator, weadopt the 9-block ResNet as the backbone network.
For the discriminator, we adopt the 3-layer
PatchGAN discriminator.
The learning rate is 0.0002 with β 1 = 0 .
5 , β 2 = 0 .
The exceptionis the cat → dog task, which requires large shape changes.
Following NEGCUT[44], we replace thepatchGAN discriminator with a 6-layer discriminator, which captures more global information.
Hyper-parameters .
We mostly follow the hyper-parameters in CUT [39].
We set L = 5 which isnumber of feature layers we use following CUT [39].
It means that we use five layers (0,4,8,12,16th)of the generator to extract patch representations.
The layers correspond to receptive fields of sizes1 × 1, 9 × 9, 15 × 15, 35 × 35, and 99 × 99.
We also use 256 patches in each layer instead of all patchesto save computation time and memory.
We set λ idt = 10 and λ density = 0 .
01 across all tasks.
Training In each iteration, we first update the density estimators by minimizing L nll .
Then weupdate the discriminator by L dis .
When optimizing the generator, to compute L patch-density , we usethe polyak averaged version of BNAF to estimate the density for patches.
For L gan , we replace thevanilla GAN loss with the LSGAN [33] objective following CUT [39].
For all dataset, we resizeimages to 256 × 256.
We keep the learning rate for the first half of the training and linearly decay it to0 in the last half of training. [Table]
Experiments
5.1 Evaluation
We follow the evaluation protocols in [54, 39, 44, 20, 50] and run all tasks once.
We report results ofmore runs in the supplementary material.
Datasets We follow the evaluation protocols in CUT [39] by running experiments on three benchmarkdatasets: label → city, cat → dog and horse → zebra.
We further run experiment on selfie → anime datasetto fully verify the effectiveness of our method.
We run label → city and horse → zebra for 400 epochs,cat → dog and selfie → anime for 200 epochs.
Metrics For the label → city task, we follow [ 39 ] and evaluate the generated photos by a pre-trainedsegmentation model DRN [48].
The DRN model translates the generated photos into segmentationlabels and then we can compare the output labels with the input labels.
We compute mean averageprecision (mAP), pixel-wise accuracy (pAcc) and average class accuracy (cAcc).
We find thatdifferent pretrained DRN model can cause huge difference in the output results, so we evaluate allmethods and report the results by our code for fair comparison.
To evaluate the visual quality of generated images, we also adopt the widely-used Frechet Inception
Distance (FID) [17] and Kernel Inception Distance (KID) [6] scores.
FID and KID both measure thedistribution divergence between the generated images and the real images.
Baselines For the first three tasks, we compare our method with different assumptions: cycleconsistency-CycleGAN [54], pairwise distance preservation – DistanceGAN and SelfDistance [5]and mutual information maximization – CUT[39], DCLGAN [16], FSeSIM [53], NEGCUT [44],
MoNCE [50] and QS-Attn [20].
For selfie → anime, we compare with CycleGAN [54], U-GAT-IT [25], CUT[39], CouncilGAN [35], ACL-GAN [52] and SpatchGAN [40].
For all tasks, we also run the Base-GAN model, i.e., λ density = 0 , to fully examine the effectiveness of our proposed regularization.
high correlation between densities in cat → dog dataset.
Our assumption states that patches of high density should be mappedto patches of high density in another domain. Therefore, it is importantto justify whether such assumption holds on various datasets. For label → city dataset, we have ground truth pairs.
For other threedatasets, we use the most recent method generations as pseudopair data, i.e., we use generation by QS-Attn on cat → dog andhorse → zebra dataset and generation by SpatchGAN on selfie → anime
[Table] [Table] dataset. Then we randomly crop the pair of images to patches onthe same location and apply PCA to reduce the dimension.
Thenwe fit a kernel density estimator for each domain.
Finally, we canobtain pair of densities for paired image patches and we compute Pearson correlation coefficient between the two sets of densities. The results are shown in Table. We can observe that p-values for all datasets are 0, suggesting that the null hypothesisthat the densities are uncorrelated can be rejected safely.
The correlation coefficients are also significantly greater than 0 and we can arrive at 0.837 on cat → dog dataset.
We also visualize theestimated log-densities of cat → dog dataset in Figure. for each dot in the figure, its coordinateis determined by log-densities of a paired image patches in two domains. We can observe a clearpositive correlation between the densities. This encouraging result well aligns with our assumption.
5.3 Comparison against Baselines
[Table]
We present the quantitative results in Table.
1 and 2.
Among the four tasks, our method achieves the bestperformance on the label → city, cat → dog tasks andselfie → anime tasks.
In particular, our method is thefirst one to improve mAP above 30 on the label → citytask.
Please note that all recent state-of-the-art methods are improved versions of CUT [39].
By contrast,we propose an effectiveness regularization which isorthogonal to the contrastive learning [39] and cycleconsistency losses [54].
Our method brings substantialimprovements over the Base-GAN model.
For the label → city task, we outperform it by 41.7% onmAP, 35.4% on pAcc and 36.5% on cAcc.
The significant improvements demonstrates that ourproposed density regularization is able to preserve important semantic information in inputs.
On thecat → dog task, our method improves the FID from 140.1 to 55.2.
On the selfie → anime task, ourmethod improves the FID from 89.5 to 80.7 and KID from 3.09 to 1.42.
The encouraging resultssuggest that our regularization is also helpful in distribution matching.
We also present the generated samples in Figure.
4 and more samples are provided in the supplementary material.
For the label → city task, we can observe that the SOTA method– QS-Attn [20] stillfaces some label flipping issue, i.e., mapping the gray area to the trees.
By contrast, our method isable to map the gray area into the building accurately.
For the cat → dog task, we can observe that the
Base-GAN model suffers the well-known mode collapse issue while our method is able to avoid it byour proposed regularization.
For the selfie → anime, our method learns to preserve the hair color ofthe first input person while keeping the headphone cable in the output image.
Unlike CouncilGAN[35] and SpatchGAN [40], our method doesn’t apply additional data augmentations, such as randombrightness and hue offset.
5.4 Analysis
Sensitivity of λ density .
Although our method only introduces one loss, it is still important to investigatethe sensitivity of the introduced hyper-parameter λ density .
We run our method on label → city task sincethe testing images are paired. The results are shown in Table. We can observe that our method
Input
QS-Attn [20]
DECENT
Truth Input
[Figure]
QS [Figure] BaseGAN
DECENT [Figure] Council
SPatchGAN BaseGAN DECENT is robust to the scale of λ .
When λ = 0 .
01 or 0 . 1 , we can achieve state-of-the-art results.
Whenλ = 0 .
0001 , it still improves the Base-GAN model from 21.86 to 28.15 on mAP metric.
When weincrease it to 0.5, which is quite far from the optimal value, we can still observe large improvementsover the Base-GAN model.
Pairwise relationship vs Density We can observe that the performance of DistanceGAN and
SelfDistance [5] are unsatisfactory in Table.
As we have argued in section 2, it may be caused byinsufficient samples.
Therefore, we build a stronger version of DistanceGAN – Patchdist.
Patchdist istrained to preserve the pairwise distance between all patch representations now.
Since many patchesare available in each iteration, the pairwise relations should be more accurate.
As shown in Table.
PatchDist can also bring improvements over Base-GAN model now (DistanceGAN and SelfDistanceare worse).
The improvements supports our argument that the neighboring information is very usefulin image translation tasks.
Our method still outperforms PatchDist by a large margin, which impliesthat the densities values are more informative than the pairwise relation in image translation tasks.
Training Time
We run all methods on NVIDIA Tesla V100-SXM2 GPU and report their trainingspeed in table 2.
One may notice that DistanceGAN [5], SelfDistance [5] and FSeSim [53] are fasterthan the Base-GAN model.
The main reason is that they don’t apply the identity regularization.
When compared to the three SOTA methods (NEGCUT [44], MoNCE [50] and QS-Attn [20]),our method achieves the best performance with the highest training speed.
NEGCUT needs togenerate negative samples with additional networks, MoNCE needs to compute the entropy of allpatch representations and MoNCE needs to address an optimal transport problem every iteration.
In contrast, we only introduce the density estimators and avoids the heavy global informationcomputation.
As a consequence, our method only needs approximately 56%, 67% and 86% trainingtime of NEGCUT [44], MoNCE [50] and QS-Attn [20], respectively.
Learned densities We visualize the learned densities in Figure.5.
The densities increase as thecolor goes from blue to red.
Firstly, we observe that our learned densities are consistent with humanjudgements and reveal the neighboring information accurately.
For example, human can easily tellthat gray patches should have more neighbors than green patches in the first row.
The densities ofgray patches shown in the second row are higher than the green patches.
Then we examine whether our method effectively enforces our assumption and whether it bringsimprovements.
On the label → city task, we observe that the mapping of Base-GAN violates ourassumption, i.e., Base-GAN maps the green patches (low density) to the building (high density)
whilegray patches (high density) to the trees (low density).
As a consequence, it suffers the label-flippingissue when compared with the ground truth photo.
In contrast, our method is able to find the correctmapping through our density regularization.
Limitation and Discussion
Although our method achieved state-of-the-art performance across many tasks, our method still suffersfrom some limitations: (1) the violation of the assumption; One may notice that the Base-GAN modelachieves the second best FID on the horse → zebra task as shown in Table.
Although our method isslightly better than other SOTA methods, it still brings some performance degradation (40.1 → 41.3).
This may be caused by the unmatched dataset statistics since it is reported that horse takes 18% pixelof the image and zebra takes 37% pixels in the dataset [39].
Therefore, patches of high density inthe horse domain may not still be of high density.
We also provide visualization and analysis inthe supplementary.
A possible solution can be adopting the attention module to only enforce ourdensity regularization on the important patches. (2) patch correspondence.
Using representation atpatch level provides more samples than image level.
But our method also faces the same limitationas patch-based methods [39, 20, 50]: the patches may not be well corresponded in some tasks.
Forexample, when the first domain is the front face while the second domain is the face profiles, weneed to learn the rotation mapping between two domains, which may cause patch-based methods tofail.
A possible solution would be considering larger patches, i.e., increase the number of layer usedto extract patch representations.
Large patches ought to contain more global information and mayaddress this issue.
In this paper, we propose a simple yet effective method to address the ill-posed unpaired image-toimage translation problem.
Departing from existing cycle and contrastive learning based methods, wepropose to preserve the neighboring information of image patches from the density persepective.
Ifthe probability density of a patch is high, it is highly likely that it has many neighbors (similar patches). Then we propose our density changing regularization where patches with high density should bemapped to patches with high density in another domain. We also propose a simple implementation toachieve this assumptions. The superior performance on various benchmark datasets demonstrate theeffectiveness of our method.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] , weprovide it in the supplemenary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A] , we follow the evaluation protocols in existing methods
and run once.
But we provide results of more runs in the supp.
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes] , we provide it in the supplementary.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] , we provide
discussions to this section in the supplementary.
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes], we present
more generated samples for better comparison.
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[Yes]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[Yes]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]