H ARMONIZING M AXIMUM L IKELIHOOD WITH GAN S FOR M ULTIMODAL C ONDITIONAL G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent advances in conditional image generation tasks, such as image-to-image translation and image inpainting, can largely be accounted to the success of conditional GAN models, which are often optimized by the joint use of the GAN loss with the reconstruction loss.
However, we show that this training recipe shared by almost all existing methods is problematic and has one critical side effect: lack of diversity in output samples.
In order to accomplish both training stability and multimodal output generation, we propose novel training schemes with a new set of losses that simply replace the reconstruction loss, and thus are applicable to any conditional generation task.
We show this by performing thorough experiments on image-to-image translation, super-resolution, and image inpainting tasks using Cityscapes, and CelebA dataset.
Quantitative evaluation also conﬁrms that our methods achieve a great diversity in outputs while retaining or even improving the quality of images.
I NTRODUCTION
Recently, active research has led to a huge progress on conditional image generation , whose typical tasks include image-to-image translation (Isola et al.
, image inpainting (Pathak et al. (2016)), super-resolution (Ledig et al.
(2017)), and video prediction (Mathieu et al.
At the core of such advances is the success of conditional GANs (Mirza & Osindero (2014)), which improve GANs by allowing the generator to take an additional code or condition to control the modes of the data being generated.
However, training GANs, including conditional GANs, is highly unstable and easy to collapse (Goodfellow et al.
To mitigate such instability, almost all previous models in conditional image generation exploit the reconstruction loss such as (cid:96) 1 / (cid:96) 2 loss in addition to the GAN loss.
Indeed, using these two types of losses is synergetic in that the GAN loss complements the weakness of the reconstruction loss that output samples are blurry and lack high-frequency structure, while the reconstruction loss offers the training stability required for convergence.
In spite of its success, we argue that it causes one critical side effect; the reconstruction loss ( e.g .
(cid:96) 1 / (cid:96) 2 loss) aggravates the mode collapse, a notorious problem of GANs.
In conditional generation tasks, which are intrinsically learning one-to-many mappings, the model is expected to generate diverse outputs from a single input, depending on some stochastic variables ( e.g .
many realistic street scene images for a single segmentation map (Isola et al., 2017)).
Nevertheless, such noise input rarely generates any diversity in the output, and surprisingly many previous methods omit a random noise source in their models.
Most papers never mention the necessity of random noise, and a few others report that the model completely ignores the noise even if it is fed into the model.
For example, Isola et al.
(2017) state that the generator simply learns to ignore the noise, and even dropout fails to incur meaningful output variation.
The objective of this paper is to propose a new set of losses that can replace the reconstruction loss with losing neither the visual ﬁdelity nor diversity in output samples.
The core idea is to use maximum likelihood estimation (MLE) loss ( e.g . (cid:96) 1 / (cid:96) 2 loss) to predict conditional statistics of the real data distribution instead of applying it directly to the generator as done in most existing algorithms.
Then, we use this prediction to assist GAN training by enforcing the statistics of the generated distribution to match the predicted statistics.
Our major contributions are three-fold.
First, we show that there is a signiﬁcant mismatch between the GAN loss and the reconstruction loss, thereby the model cannot achieve the optimality w.r.t.
both losses.
Second, we propose novel training schemes named MLMM and MCMLE that enable the model to accomplish both training stability and multimodal output generation.
Our methods simply replace the reconstruction loss, and thus are applicable to any conditional generation task.
Finally, we show the effectiveness and generality of our methods through extensive experiments on three generation tasks.
Our methods outperform baselines in terms of realism and diversity.
R ELATED W ORKS
Conditional Generation Tasks.
Since the advent of GANs (Goodfellow et al.
and conditional GANs (Mirza & Osindero (2014)), there has been a large body of work in conditional generation tasks.
A non-exhaustive list includes image translation (Isola et al., 2017; Wang et al., 2017; 2018; Zhang et al., 2017b;
Lyu et al., 2017; Sangkloy et al., 2017; Xian et al., 2017; Zhang et al., 2017a;
Liu et al., 2017), image inpainting (Pathak et al., 2016; Iizuka et al., 2017; Ouyang et al., 2018; Olszewski et al., 2017; Yu et al., 2018; Sabini & Rusak, 2018; Song et al., 2018; Li et al., 2017; Yang et al., 2018; Yeh et al., 2017), super-resolution (Ledig et al., 2017; Xie et al., 2018; Mahapatra et al., 2017; Xu et al., 2017; Bulat et al., 2018; Sajjadi et al., 2017), video prediction (Mathieu et al., 2016; Jang et al., 2018;
Lu et al., 2017; Villegas et al., 2017; Zhou & Berg, 2016; Bhattacharjee & Das, 2017; Vondrick & Torralba, 2017).
However, existing models have one common limitation: lack of stochasticity for diverse output.
In spite of the fact that the tasks are to be one-to-many mapping, they ignore random noise input which is necessary to generate diverse samples from a single input.
A number of works such as (Mathieu et al., 2016; Isola et al., 2017; Xie et al., 2018) have tried injecting random noise into their models but discovered that the models discard it and instead learn a deterministic mapping.
Multimodality Enhancing Models.
It is not fully understood yet why conditional GAN models fail to learn the full multimodality of data distribution.
Recently, there has been a series of attempts to incorporate stochasticity in conditional generation as follows.
(1) Conditional VAE-GAN.
VAE-GAN (Larsen et al., 2016) is a hybrid model to combine the decoder in VAE (Kingma & Welling, 2014) with the generator in GAN (Goodfellow et al., 2014).
Its conditional variants have been also proposed such as CVAE-GAN (Bao et al., 2017), BicycleGAN (Zhu et al., 2017) and SAVP (Lee et al., 2018a).
These models harness the strengths of the two models, output ﬁdelity by GANs and diversity by VAEs and can produce a wide range of realistic images.
Intuitively, the VAE structure drives the generator to exploit latent variables to represent multimodality of the conditional distribution.
(2) Disentangled representation.
Huang et al.
and Lee et al.
2018b) propose to learn disentangled representation for multimodal unsupervised image-to-image translation.
These models split the embedding space into a domain-invariant space for sharing information across domains and a domain-speciﬁc space for capturing styles and attributes.
The models encode an input to the domain-invariant embedding and sample domain-speciﬁc embedding from some prior distribution.
By feeding the two embeddings into the decoder of the target domain, the model can generate diverse samples in a target domain.
Conditional VAE GANs and disentangling-based methods both leverage the latent variable to prevent the model from discarding the multimodality of output samples.
In this paper, we present a simpler and orthogonal direction to achieve multimodal conditional generation by introducing novel loss functions that can replace the reconstruction loss
L OSS M ISMATCH OF C ONDITIONAL GAN S
We brieﬂy review the objective of conditional GANs in section 3.1 and discuss why the two loss terms cause loss of modality in the sample distribution of the generator in section 3.2.
3. P RELIMINARY : O BJECTIVE OF C ONDITIONAL GAN S
The goal of conditional GANs is to learn to generate samples that are indistinguishable from real data for a given input.
The objective of conditional GANs usually consists of two terms, the GAN loss L GAN and the reconstruction loss Rec .
[Equation]
Another popular loss term is the perceptual loss (Johnson et al., 2016; Bruna et al., 2016; Ledig et al., 2017).
While the reconstruction loss encodes the pixel-level distance, the perceptual loss is deﬁned as the distance between the features encoded by neural networks.
Since they share the same form ( e.g . (cid:96) 1 /(cid:96) 2 loss), we consider the perceptual loss as a branch of the reconstruction loss.
The loss L GAN is deﬁned to minimize some distance measure ( e.g . JS-divergence) between the true and generated data distribution conditioned on input x .
The training scheme is often formulated as the following minimax game between the discriminator D and the generator G .
[Equation]
where each data point is a pair ( x, y ) , and G generates outputs given an input x and a random noise z .
Note that the discriminator observes x , which are crucial for the performance (Isola et al., 2017).
The most widely used reconstruction losses in conditional GAN literature are the (cid:96) 1 (Isola et al., 2017;
Wang et al., 2017) and (cid:96) 2 loss (Pathak et al., 2016; Mathieu et al., 2016).
Both losses can be formulated as follows with p
= 1 , 2 respectively.
[Equation]
These two losses naturally stem from the maximum likelihood estimations (MLEs) of the parameters of Laplace and Gaussian distribution, respectively.
The likelihood of dataset ( X , Y ) assuming each distribution is deﬁned as follows.
[Equation]
where N is the size of the dataset, and f is the model parameterized by θ .
The central and dispersion measure for Gaussian are the mean and variance σ 2 , and the correspondences for Laplace are the median and mean absolute deviation (MAD) b .
Therefore, using (cid:96) 2 loss leads the model output f θ ( x ) to become an estimate of the conditional average of y given x , while using (cid:96) 1 loss is equivalent to estimating the conditional median of y given x (Bishop, 2006).
Note that the model f θ is trained to predict the mean (or median) of the data distribution, not to generate samples from the distribution.
3. L OSS OF M ODALITY BY THE R ECONSTRUCTION L OSS
We argue that the joint use of the reconstruction loss with the GAN loss can be problematic, because it may worsen the mode collapse.
Below, we discuss this argument both mathematically and empirically.
We analyze the mismatch between GAN loss and (cid:96) 2 loss in terms of the set of optimal generators.
Refer to appendix F for the case of (cid:96) 1 loss. The sets of optimal generators for the GAN loss and the (cid:96) 2 loss, denoted by G and R respectively, can be formulated as follows: [Equation]
The problem of (cid:96) 2 loss is that it forces the model to predict only the mean of p ( y | x ) , while pushing the conditional variance to zero. According to James (2003), the (cid:96) 2 loss can be decomposed into one irreducible term, the inherent noise of data, and two reducible terms: the squared bias and the variance of outputs.
Thus, (cid:96) 2 loss is minimized when both bias and variance are zero, which implies that R is a subset of V where G has no conditional variance.
[Equation]
In conditional generation tasks, however, many outputs can be paired with the input, i.e . the conditional variance Var( y | x ) is non-zero. Thereby we conclude G ∩ V = ∅ , which reduces to G ∩ R = ∅ . It means that any generator G cannot be optimal to both GAN and L2 loss simultaneously.
Therefore, it is hard to anticipate what solution is attained by training and how the model behaves when the two losses are combined. One thing for sure is that the mode collapse is an inevitable consequence.
[Figure] Figure 1 shows the actual examples of the Pix2Pix model (Isola et al.
(2017)) applied to translate segmentation labels to realistic photos in Cityscapes dataset (Cordts et al., 2016).
We slightly modify the model so that it takes additional noise input.
We use the (cid:96) 1 loss as the reconstruction loss as it was done in the original paper.
We train four models that use different combinations of loss terms, and generate four samples with different noise input.
As shown in Figure 1(c), the model trained with only the GAN loss fails to generate realistic images, since the signal from the discriminator is too unstable to learn the translation task.
In Figure 1(d), the model with only the (cid:96) 1 loss is more stable but produces results far from being realistic.
The combination of the two losses in Figure 1(e) helps not only to reliably train the model but also to generate visually appealing images; yet, it’s results lack variation.
This phenomenon is also reported in (Mathieu et al., 2016; Isola et al., 2017), although the cause is unknown.
Pathak et al.
(2016) and Iizuka et al.
(2017) even state that better results are obtained without noise in the models.
Finally, Figure 1(f) shows that our new objective enables the model to generate not only visually appealing but also diverse output samples.
A PPROACH
We propose novel alternatives of the reconstruction loss that are applicable to virtually any conditional generation task.
Trained with our new loss terms, the model can accomplish both the stability of training and multimodal generation as already seen in Figure 1(f).
Figure 2 illustrates architectural comparison between conventional conditional GANs and our models.
In the conventional conditional GAN, the MLE losses are applied to the generator’s objective to make sure that it generates an output sample well matched to its ground-truth.
On the other hand, our key idea is to apply
MLE losses to make sure that the statistics , such as mean or variance, of the conditional distribution p ( y | x ) are similar between the generator’s sample distribution and actual data distribution.
In section 4.1, we extend the (cid:96) 1 / (cid:96) 2 loss to estimate all the parameters of the Laplace and Gaussian distribution.
In section 4.2 and 4.3, we present two novel training schemes for conditional GANs.
Finally, in section 4.4, we discuss our methods’ novelties over existing algorithms.
4. T HE MLE FOR M EAN AND V ARIANCE
The (cid:96) 2 loss encourages the model to perform the MLE of the conditional mean of y given x while the variance σ 2 , the other parameter of Gaussian, is assumed to be ﬁxed.
If we allow the model to
[Figure]
𝐺 𝐷 𝑃 ℒ GAN ෤𝜎 2 ෤𝜇 ො𝜇, ො𝜎ℒ MLMM ℒ MLE 𝑥 𝑦 ෤𝑦
1:𝐾
𝑧 1:𝐾 (b) MLMM 𝐺 𝐷 ℒ GAN ෤𝜎 2 ෤𝜇 𝑥 ෤𝑦 1:𝐾 ℒ MCMLE 𝑦 𝑧 1:𝐾 (c) MCMLE estimate the conditional variance as well, the MLE loss corresponds to
[Equation]
where ˆ µ, ˆ σ 2 = f θ ( x ) (6) where the model f θ now estimates both ˆ µ and ˆ σ 2 for x .
Estimating the conditional variance along with the mean can be interpreted as estimating the heteroscedastic aleatoric uncertainty in Kendall & Gal (2017) where the variance is the measure of aleatoric uncertainty.
For the Laplace distribution, we can derive the similar MLE loss as
[Equation]
where ˆ m is the predicted median and ˆ b is the predicted MAD (Bloesch et al., 2018).
In practice, it is more numerically stable to predict the logarithm of variance or MAD (Kendall & Gal, 2017).
In the following, we will describe our methods mainly with the (cid:96) 2 loss under Gaussian assumption.
It is straightforward to obtain Laplace versions of our methods for the (cid:96) 1 loss by simply replacing the mean and variance with the median and MAD.
4. M AXIMUM L IKELIHOOD M OMENT M ATCHING (MLMM)
Our ﬁrst model is named as Maximum Likelihood Moment Matching (MLMM) whose overall architecture is depicted in Figure 2(b).
Its architecture follows that of a conditional GAN, but there are two updates.
First, the generator produces K different samples ˜
y 1: K for each input x by varying noise input z .
Second, we introduce a separate component predictor , which is a clone of the generator with some minor differences: (i) no noise source as input, and (ii) both mean and variance prediction as output.
The predictor uses the MLE loss in Eq.(6) with ground-truth y to obtain the predictions of conditional mean and variance, i.e . ˆ µ and ˆ σ 2 .
When training the generator, we utilize the predicted statistics of real distribution to guide the outputs of the generator.
Speciﬁcally, we match the predicted mean/variance and the mean/variance of the generator’s distribution, which is computed by the sample mean ˜ µ and variance ˜ σ 2 from ˜ y 1: K .
Then, we deﬁne the MLMM loss L MLMM as the sum of squared errors between predicted statistics and sample statistics.
The ﬁnal loss becomes the weighted sum of GAN loss and MLMM loss.
In summary,
[Equation]
i = ˜
y i , ˜
σ 2 =
K −
K (cid:88) i =
(˜ y i − ˜ µ ) 2 ,
where ˜ y 1: K = G ( x, z 1: K ) .
One possible variant is to match only the ﬁrst moment µ .
We denote the original method MLMM 1 /and the variant MLMM 1 where the number indicates the order of matched moments.
In addition, we can easily derive the Laplace version of MLMM 1 / 2 that uses median and MAD.
4. M ONTE C ARLO MLE
In MLMM, we use a surrogate entity, predictor, to predict the conditional moments and to match them with the sample moments.
Another approach may be directly applying MLE losses to the sample statistics; since the generator is supposed to approximate the real distribution, we can consider the statistics of the generated distribution as the estimators for the statistics of the real distribution.
Like MLMM, we can use the sample statistics ˜ µ and ˜ σ 2 as approximated statistics of the generated distribution (in this case, it corresponds to Monte Carlo estimation).
Then we replace ˆ µ and ˆ σ 2 in
Eq.(6) with ˜ µ and ˜ σ 2 to obtain the Monte Carlo MLE (MCMLE) loss:
[Equation]
where ˜ µ and ˜ σ 2 are deﬁned as Eq.(9).
The MCMLE architecture is illustrated in Figure 2(c).
Similarly to MLMM, we can build a variant MCMLE 1 , where only the (cid:96) 2 loss between y and ˜ µ is used.
Also, Laplace assumption yields the Laplace versions of MCMLE 1 / 2 .
The detailed algorithms of all eight variants are presented in appendix A.
Compared to MLMM, MCMLE allows the generator to access real data y directly, thus there is no bias caused by the predictor.
On the other hand, the use of predictor in MLMM provides less variance in target values and leads more stable training especially when batch or sample size is small.
Another important aspect worth comparison is overﬁtting, which should be carefully considered when using MLE with ﬁnite training data.
In MLMM, we can choose the predictor with the smallest validation loss to avoid overﬁtting, and freeze it while training the generator.
Therefore, the generator trained with MLMM suffers less from overﬁtting, compared to the generator trained with MCMLE which directly observes training data.
To sum up, the two methods have their own pros and cons.
We will empirically compare the behaviors of these two approaches in section 5.2.
4. A NALYSES ON O UR A PPROACH
We discuss why our approach does not suffer from a loss of diversity in the output samples unlike existing models using the reconstruction loss in terms of the set of optimal generators, as in section 3.2.
The sets of optimal mappings for our loss terms are formulated as follows:
[Equation]
where M 1 corresponds to MLMM 1 and MCMLE 1 while M 2 corresponds to MLMM 1 / 2 and
MCMLE 1 / 2 .
It is straightforward to show that G ⊂ M 2 ⊂ M 1 ; if p data ( y | x ) = p G ( y | x ) is satisﬁed at optimum, the conditional expectations and variations of both sides should be the same too:
[Equation] [Equation]
To summarize, there is no generator that is both optimal for the GAN loss and the (cid:96) 2 loss since G ∩ R = ∅ .
Moreover, as the (cid:96) 2 loss pushes the conditional variance to zero, the ﬁnal solution is likely to lose multimodality.
On the other hand, the optimal generator w.r.t.
the GAN loss is also optimal w.r.t.
our loss terms since G ⊂ M 2 ⊂ M 1 .
This proof may not fully demonstrate why our model does not give up multimodality, which could be an interesting future work in line with the mode collapsing issue of GANs.
Nonetheless, we can at least assert that our loss functions do not suffer from the same side effect as the reconstruction loss does.
Another remark is that this proof is conﬁned to conditional GAN models without explicit encoding of latent variable.
It may not be applicable to the models that explicitly encode latent variables such as Bao et al.
(2017), Zhu et al.
(2017), Lee et al. (2018a), Huang et al. (2018), and Lee et al.
(2018b), since the latent variables provide meaningful information about the target.
[Figure]  E XPERIMENTS In order to show the generality of our methods, we apply them to three conditional generation tasks: image-to-image translation, super-resolution and image inpainting, for each of which we select as base models Pix2Pix, SRGAN (Ledig et al., 2017), and GLCIC (Iizuka et al., 2017), respectively.
We use the Maps dataset (Isola et al., 2017) and the Cityscapes dataset (Cordts et al., 2016) for image translation and the CelebA dataset (Liu et al., 2015) for the other tasks.
We minimally modify the base models to include random noise source and train them with the MLMM 1 / 2 objective.
We do not use any other loss terms such as perceptual loss, and the models are trained from scratch.
We present all the training and implementation details and more thorough results in the appendix.
5. Q UALITATIVE E VALUATION
In every task, most of our methods successfully generate diverse images as presented in Figure 3.
Our methods are generally applicable to a wide variety of conditional generation tasks.
From qualitative aspects, the most noticeable difference between MLMM and MCMLE lies in the training stability.
We ﬁnd that MLMM works as well as reconstruction loss in all three tasks. On the other hand, we could not ﬁnd any working conﬁguration of MCMLE in the image inpainting task.
Also, MCMLE is more sensitive to the number of samples generated for each input.
In SRGAN experiments, for example, both methods converge reliably with the sampling number of 24, while MCMLE often diverges at the sampling number of 12.
Although MCMLE is simpler and can be trained in an end-to-end manner, its applicability is rather limited compared to MLMM.
We provide generated samples for each conﬁguration in the appendix.
5. Q UANTITATIVE E VALUATION
As done in Zhu et al.
, we quantitatively measure diversity and realism of generated images.
We evaluate our methods on Pix2Pix–Cityscapes, SRGAN–CelebA, and GLCIC–CelebA tasks.
For each (method, task) pair, we generate 20 images from each of the 300 different inputs using the trained model.
As a result, the test sample set size is 6,000 in total.
For diversity, we measure the average LPIPS score (Zhang et al., 2018).
Among 20 generated images per input, we randomly choose 10 pairs of images and compute conditional LPIPS values.
We then average the scores over the test set.
For realism, we conduct a human evaluation experiment from 33 participants.
We present a real or fake image one at a time and ask participants to tag whether it is real or fake.
The images are presented for 1 second for SRGAN/GLCIC and 0.5 second for Pix2Pix, similarly to Zhu et al.
We calculate the accuracy of identifying fake images with averaged F-measure F and use 2(1 − F ) as the realism score.
The score is assigned to 1.0 when all samples are completely indistinguishable from real images and 0 when evaluators make no misclassiﬁcation. [Table]
where F is the average of F-measures of identifying fake by human evaluators.
Diversity is scored by the average of conditional LPIPS values.
In both metrics, higher are better.
In all three tasks, our methods generate highly diverse images with competitive or even better realism.
There are eight conﬁgurations of our methods in total, depending on the MLE (Gaussian and
Laplace), the number of statistics (one and two) and the method type (MLMM and MCMLE).
We test all variants for the Pix2Pix task and only Gaussian MLMM 1 / 2 for the others.
We compare with base models in every task and additionally BicycleGAN (Zhu et al., 2017) for the Pix2Pix task.
Since BicycleGAN has not been applied to image inpainting and super-resolution, we do not report its result on the two tasks.
We use the ofﬁcial BicycleGAN implementation from the authors with minimal modiﬁcation.
Table 1 summarizes the results.
In terms of both realism and diversity, our methods achieve competitive performance compared to the base models, sometimes even better.
For instance, in Pix2Pix–Cityscapes setting, three of our methods, Gaussian MLMM 1 , MLMM 1 / 2 , and MCMLE 1 , signiﬁcantly outperform BicycleGAN and Pix2Pix+noise in both measures.
Without exception, the diversity scores of our methods are far greater than those of the baselines while maintaining competitive realism scores.
These results conﬁrm that our methods generate a broad spectrum of quality images from a single input.
Interestingly, MLMM 1 and MCMLE 1 generate comparable or even better outputs compared to MLMM 1 / 2 and MCMLE 1 / 2 .
That is, matching means could be enough to guide GAN training in many cases.
It implies that adding more statistics ( i.e . adding more guidance signal to the model) may be helpful in some cases, but generally may not improve the performance.
Moreover, additional errors may arise from predicting more statistics, which could misguide the GAN training.
C ONCLUSION
In this work, we pointed out that there is a signiﬁcant mismatch between conventional reconstruction losses and GAN loss.
As alternatives, we proposed a set of novel loss functions named MLMM and MCMLE that enable conditional GAN models to accomplish both stability of training and multimodal generation.
Empirically, we showed that our loss functions were successfully integrated with multiple state-of-the-art models for image translation, super-resolution, and image inpainting tasks, for which our method generated realistic image samples of high visual ﬁdelity and variability on Cityscapes and CelebA datasets.
There are numerous possible directions beyond this work.
First, there are other conditional generation tasks that we did not cover, such as text-to-image synthesis, text-to-speech synthesis, and video prediction, for which our methods can be directly applied to generate diverse, high-quality samples.
Second, in terms of statistics matching, our methods can be extended to explore other higher order statistics or covariance.
Third, using the statistics of high-level features may capture correlations which are unreachable with pixel-level statistics.
A A LGORITHMS
We elaborate on the algorithms of all eight variants of our methods in detail from Algorithm 1 to
Algorithm 8.
The presented algorithms assume a single input per update, although we use mini-batch training in practice.
Also we use non-saturating GAN loss, − log D ( x, ˜ y )
(Goodfellow, 2016).
For Laplace MLEs, the statistics that we compute are median and MAD.
Unlike mean, however, the gradient of median is deﬁned only in terms of the single median sample.
Therefore, a naive implementation would only calculate the gradient for the median sample, which is not effective for training.
Therefore, we use a special trick to distribute the gradients to every sample.
In MLMM, we ﬁrst calculate the difference between the predicted median and the sample median, and then add it to samples ˜ y 1: K to set the target values t 1: K .
We consider t 1: K as constants so that the gradient is not calculated for the target values (this is equivalent to Tensor.detach() in PyTorch and tf.stop gradient() in TensorFlow).
Finally, we calculate the loss between the target values and samples, not the medians.
We use the similar trick for MCMLE.
Require:
Generator G , discriminator D , pre-trained predictor P , MLMM loss coefﬁcient λ MLMM
Require: input x , ground truth y , the number of samples K 1: for i = 1 to K do 2:
z i ← GenerateNoise() 3: ˜
y
i ← G ( x, z i ) { Generate K samples } 4: end for 5: ˆ µ ← P ( x ) { Predicted mean } 6: ˜ µ ← 1 K (cid:80) Ki =
1 ˜
y i { Sample mean } 7: L MLMM ← (ˆ µ − ˜ µ )8:
L GAN ← 1 K (cid:80) Ki
=1 − log D ( x, y i ) 9: θ G ← Optimize ( θ G , ∇ θ G L GAN + λ MLMM L MLMM )
Require:
Generator G , discriminator D , pre-trained predictor P , MLMM loss coefﬁcient λ MLMM
Require: input x ground truth y
the of samples do
GenerateNoise
Require:
Generator G , discriminator D , pre-trained predictor P , MLMM loss coefﬁcient λ MLMM
Require: input x , ground truth y , the number of samples K 1: for i = 1 to K do 2:
z i ← GenerateNoise() 3: ˜
y
i ← G ( x, z i ) { Generate K samples } 4: end for 5: ˆ m ← P ( x ) { Predicted median } 6: ˜ m ← med(˜ y 1: K ) { Sample median } 7: t 1: K = detach(˜
y
: K + ( ˆ m − ˜ m )) 8: L MLMM ← 1 K (cid:80) Ki
=
( t i − ˜ y i )9: L GAN ← 1 K (cid:80)
Ki
=1 − log D ( x, y i ) 10: θ G ← Optimize ( θ G , ∇ θ G L GAN + λ MLMM
L MLMM )
Require:
Generator G , discriminator D , pre-trained predictor P , MLMM loss coefﬁcient λ MLMM
Require: input x , ground truth y , the number of samples K 1: for i = 1 to K do 2:
z i ← GenerateNoise() 3: ˜
y
i ← G ( x, z i ) { Generate K samples } 4: end for 5: ˆ m, ˆ b ← P ( x ) { Predicted median and MAD } 6: ˜ m ← med(˜ y 1: K ) { Sample median } 7: ˜
b ← 1 K (cid:80) Ki =1 | ˜
y i − ˜ m | { Sample MAD } 8: t 1: K = detach(˜ y 1: K
+ ( ˆ m − ˜ m )) 9: L MLMM ← 1 K (cid:80) Ki
=
( t i − ˜ y i )
+ (ˆ b − ˜ b )10: L GAN ← 1 K (cid:80)
Ki
=1 − log D ( x, y i ) 11: θ G ← Optimize ( θ G , ∇ θ G L GAN + λ MLMM
L MLMM )
Require:
Generator G , discriminator D , MCMLE loss coefﬁcient λ MCMLE
Require: input x , ground truth y , the number of samples K 1: for i = 1 to K do 2:
z i ← GenerateNoise() 3: ˜
y
i ← G ( x, z i ) { Generate K samples } 4: end for 5:
˜ µ ← 1 K (cid:80) Ki =
1 ˜
y i { Sample mean } 6: L MCMLE ← ( y − ˜ µ )7:
L GAN ← 1 K (cid:80) Ki
=1 − log D ( x, y i ) 8: θ G ← Optimize ( θ G , ∇ θ G L GAN + λ MCMLE L MCMLE )
Require:
Generator G , discriminator D , MCMLE loss coefﬁcient λ MCMLE
Require: input x , ground truth y , the number of samples K 1: for i = 1 to K do 2:
z i ← GenerateNoise() 3: ˜
y
i ← G ( x, z i ) { Generate K samples } 4: end for 5:
˜ µ ← 1 K (cid:80) K i =
˜ y
i { Sample mean } 6: ˜ σ 2 ← 1 K − 1 (cid:80) Ki
=
(˜ y i − ˜ µ ) 2 { Sample variance } 7: L MCMLE ← ( y − ˜ µ ) 2 2˜ σ+ 12 log ˜ σ8: L GAN ← 1 K (cid:80)
Ki
=1 − log D ( x, y i ) 9: θ G ← Optimize ( θ G , ∇ θ G L GAN + λ MCMLE L MCMLE )
Require:
Generator G , discriminator D , MCMLE loss coefﬁcient λ MCMLE
Require: input x , ground truth y , the number of samples K 1: for i = 1 to K do 2:
z i ← GenerateNoise() 3: ˜
y
i ← G ( x, z i ) { Generate K samples } 4: end for 5: ˜ m ← med(˜ y 1: K ) { Sample median } 6: t 1: K = detach(˜
y
: K + ( y − ˜ m )) 7: L MCMLE ← 1 K (cid:80)
Ki =
| t i − ˜
y i | 8: L GAN ← 1 K (cid:80) Ki
=1 − log D ( x, y i ) 9: θ G ← Optimize ( θ G , ∇ θ G L GAN + λ MCMLE L MCMLE )
Require:
Generator G , discriminator D , MCMLE loss coefﬁcient λ MCMLE
Require: input x , ground truth y , the number of samples K 1: for i = 1 to K do 2:
z i ← GenerateNoise() 3: ˜
y
i ← G ( x, z i ) { Generate K samples } 4: end for 5: ˜ m ← med(˜ y 1: K ) { Sample median } 6: ˜
b ← 1 K (cid:80) Ki =1 | ˜ y
i − ˜ m | { Sample MAD } 7: t 1: K = detach(˜
y 1: K
+ ( y − ˜ m )) 8: L MCMLE ← 1 K (cid:80)
Ki =
| t i − ˜
y
i | ˜ b + log˜ b 9: L GAN ← 1 K (cid:80) Ki
=1 − log D ( x, y i ) 10: θ G ← Optimize ( θ G , ∇ θ G L GAN + λ MCMLE L MCMLE )
B T RAINING D ETAILS B. C OMMON C ONFIGURATIONS
We use PyTorch for the implementation of our methods.
In every experiment, we use AMSGrad optimizer (Reddi et al., 2018) with LR = 10 − 4 , β 1 = 0 .
5 , β 2 = 0 .
We use the weight decay of a rate 10 − 4 and the gradient clipping by a value 0 .
In case of MLMM, we traine the predictor until it is overﬁtted, and use the checkpoint with the lowest validation loss.
The weight of GAN loss is ﬁxed to 1 in all cases.
Convergence speed .
Our methods need more training steps (about 1.5 × ) to generate high-quality images compared to those with the reconstruction loss.
This is an expectable behavior because our methods train the model to generate a much wider range of outputs.
Training stability . MLMM is similar to the reconstruction loss in terms of training stability. Our methods work stably with a large range of hyperparameter λ . For example, the loss coefﬁcient of
MLMM can be set across several orders of magnitude (from tens to thousands) with similar results.
However, as noted, MCMLE is unstable compared to MLMM.
B. P IX 2P IX
Our Pix2Pix variant is based on the U-net generator from https github. com/ junyanz/pytorch-CycleGAN-and-pix2pix .
• Noise input: We concatenate Gaussian noise tensors of size H × W × 32 at the 1 × 1 , 2 × 2 , 4 × 4 feature map of the decoder.
Each element in the noise tensors are independently sampled from N (0 , 1) .
• Input normalization: We normalize the inputs so that each channel has a zero-mean and a unit variance.
• Batch sizes: We use 16 for the discriminator and the predictor and 8 for the generator.
When training the generator, we generate 10 samples for each input, therefore its total batch size is 80.
• Loss weights: We set λ MLMM =
λ MCMLE = 10 .
For the baseline, we use (cid:96) 1 loss as the reconstruction loss and set λ (cid:96) 1 = 100 .
• Update ratio: We update generator once per every discriminator update.
B. SRGAN
Our SRGAN variant is based on the PyTorch implementation of SRGAN from https://github.com/zijundeng/SRGAN .
• Noise input: We concatenate Gaussian noise tensor of size H × W × 16 at each input of the residual blocks of the generator, except for the ﬁrst and last convolution layers.
Each element in the noise tensors are independently sampled from N (0 , 1) .
• Input normalization: We make 16 × 16 × 3 input images’ pixel values lie between -1 and 1.
We do not further normalize them with their mean and standard deviation.
• Batch sizes: We use 32 for the discriminator and the predictor and 8 for the generator.
When training the generator, we generate 24 samples for each input, and thus its total batch size is 192.
• Loss weights: We set λ MLMM =
2400 and λ MCMLE = 20 .
For the baseline, we use (cid:96) 2 loss as the reconstruction loss and set λ (cid:96) 2 = 1000 .
• Update ratio: We update generator ﬁve times per every discriminator update.
B. GLCIC
We built our own PyTorch implementation of the GLCIC model.
• Noise input: We concatenate Gaussian noise tensor of size H × W × 32 at each input of the ﬁrst and second dilated convolution layers.
We also inject the noise to the convolution layer before the ﬁrst dilated convolution layer.
Each element in the noise tensors are independently sampled from N (0 , 1) .
• Input resizing and masking: We use square-cropped CelebA images and resize them to 128 × 128 .
For masking, we randomly generate a hole of size between 48 and 64 and ﬁll it with the average pixel value of the entire training dataset.
• Input normalization: We make 128 × 128 × 3 input images’ pixel values lie between 0 and 1.
We do not further normalize them with their mean and standard deviation.
• Batch sizes: We use 16 for the discriminator and the predictor and 8 for the generator.
When training the generator, we generate 12 samples for each input, therefore its total batch size is 96.
• Loss weights: For GLCIC, we tested Gaussian MLMM 1 / 2 and MCMLE methods.
We successfully trained GLCIC with Gaussian MLMM 1 / 2 using λ MLMM =
However, we could not ﬁnd any working setting for MCMLEs.
For the baseline model, we use (cid:96)loss for the reconstruction loss and set λ (cid:96) 2 = 100 .
• Update ratio: We update generator three times per every discriminator update.
C P REVENTIVE E FFECTS FOR M ODE C OLLAPSE
Our MLMM and MCMLE methods have preventive effect on mode collapse.
Figure 4 shows toy experiments of unconditional generation on synthetic 2D data which is hard to learn with GANs due to mode collapse.
We train a simple 3-layer MLP with different objectives.
When trained only with GAN loss, the model captures only one mode as shown in ﬁgure 4b.
Adding (cid:96) 2 loss cannot ﬁx this issue either as in ﬁgure 4c.
In contrast, all four of our methods (ﬁgure 4e-4h) prevent mode collapse and successfully capture all eight modes.
Notice that even the simpler variants MLMMand MCMLE 1 effectively keep the model from mode collapse.
Intuitively, if the generated samples are biased toward a single mode, their statistics, e.g . mean or variance, deviate from real statistics.
Our methods penalize such deviations, thereby reducing mode collapse signiﬁcantly.
Although we restrict the scope of this paper to conditional generation tasks, these toy experiments shows that our methods has a potential to mitigate mode collapse and stabilize training even for unconditional generation tasks.
[Figure]
D G ENERATED S AMPLES D.
I MAGE - TO -I MAGE T RANSLATION (P IX 2P IX )
Generated Samples Generated Samples
Samplemean/var
Gaussian MLMM
Laplace MLMM
Base+noise
Laplace MLMM
Gaussian MLMM
Base+noise
Input/GT Input/GT
Predicted mean/var
Samplemean/var
Predicted mean/var
Input/GT Input/GT
Predicted mean/var Predicted mean/var
Generated Samples Generated Samples
Samplemean/var Samplemean/var
Generated Samples Generated Samples
Samplemean/var
Gaussian MLMM
Laplace MLMM
Base+noise
Samplemean/var
Input/GT Input/GT
Predicted mean/var Predicted mean/var
[Figure]
D. S UPER -R ESOLUTION (SRGAN)
The ﬁrst rows of following images are composed of input, ground-truth, predicted mean, sample mean, predicted variance and sample variance.
The other rows are generated samples.
D. I MAGE I NPAINTING (GLCIC)
This section shows the results of GLCIC–
CelebA task.
The images are shown in the same manner as the previous section.
E D ECOMPOSITION OF R ECONSTRUCTION L OSS
According to James (2003), for any symmetric loss function L s and an estimator ˆ y for y , the loss is decomposed into one irreducible term Var( y ) and two reducible terms: SE (ˆ y, y ) and VE (ˆ y, y ) , where SE refers to systematic effect , the change in error caused by bias, while VE refers to variance effect , the change in error caused by variance.
[Equation]
S is an operator that is deﬁned to be
[Equation]
arg
[Equation]
[ L s ( y, µ )] .
Notice that the total loss is minimized when ˆ y =
S
ˆ y =
Sy reducing both SE and VE to 0.
For (cid:96)loss, Sy and S ˆ y are the expectations of y and ˆ y .
[Equation]
Also, SE and VE correspond to the squared bias and the variance, respectively.
[Equation] This decomposition offers another interpretation of the (cid:96) 2 loss: minimizing the bias and the variance of prediction.
F M ISMATCH BETWEEN (cid:96) 1 L OSS AND GAN L OSS
In section 4.4, we showed that if there exists some x such that Var( y | x ) > 0 , there is no generator optimal for both (cid:96) 2 loss and GAN loss.
On the other hand, (cid:96) 1 loss has some exceptional cases where the generator can minimize GAN loss and (cid:96) 1 loss at the same time.
We identify what the cases are and explain why such cases are rare.
To begin with, (cid:96) 1 loss is decomposed as follows:
[Equation]
To minimize (cid:96) 1 loss, we need the gradient w.r.t.
G ( x, z ) to be zero for all x and z that p ( x ) > 0 and p ( z ) > 0 . Note that this is a sufﬁcient condition for minimum since the (cid:96) 1 loss is convex.
[Equation]
Therefore, G ( x, z ) should be the conditional median to minimize the loss.
Unlike (cid:96) 2 loss, there can be an interval of G ( x, z ) that satisﬁes Eq.(17).
Speciﬁcally, any value between interval [ a, b ] is a conditional median if (cid:82) a −∞ p ( y | x ) dy = (cid:82) b
−∞ p ( y | x ) dy = 12 .
If every real data belongs to the interval of conditional median, then the generator can be optimal for both GAN loss and (cid:96) 1 loss.
For instance, assume that there are only two discrete values of y possible for any given x , say −and 1 , with probability 0 .
5 for each.
Then the interval of median becomes [ − 1 , 1] , thus any G ( x, z ) in the interval [ − 1 , 1] minimizes the (cid:96) 1 loss to 1 .
If the generated distribution is identical to the real distribution, i.e . generating − 1 and 1 with the probability of 0 . 5 , the generator is optimal w.r.t.
both
GAN loss and (cid:96) 1 loss.
However, we note that such cases hardly occur.
In order for such cases to happen, for any x , every y with p ( y | x ) > 0 should be the conditional median, which is unlikely to happen in natural data such as images.
Therefore, the set of optimal generators for (cid:96) 1 loss is highly likely to be disjoint with the optimal set for GAN loss.
G E XPERIMENTS ON M ORE C OMBINATIONS OF L OSS F UNCTIONS
We present some more experiments on different combinations of loss functions.
The following results are obtained in the same manner as section D.2 and section D.3.
Figure 11 shows the results when we train the Pix2Pix model only with our loss term (without the
GAN loss).
The samples of MLMM 1 and MCMLE 1 are almost similar to those with the reconstruction loss only.
Since there is no reason to generate diverse images, the variances of the samples are near zero while the samples are hardly distinguishable from the output of the predictor.
On the other hand, MLMM 1 / 2 and MCMLE 1 / 2 do generate diverse samples, although the variation styles are different from one another.
In MLMM 1 / 2 , where the attraction for diversity is relatively mild, the samples show low-frequency variation.
Intriguingly, MCMLE 1 / 2 incurs high-frequency variation patterns as the sample variance of MCMLE 1 / 2 is much closer than MLMM 1 / 2 to the predicted variance.
[Figure]
[Figure]
Another experiment that we carry out is about the joint use of all loss terms: GAN loss, our losses and reconstruction loss.
Speciﬁcally, we use the following objective with varying λ
Rec from 0 to 100 to train the Pix2Pix model on Cityscapes dataset.
[Figure]
Figure 12 shows the results of the experiments.
As λ Rec increases, the sample variance reduces.
This conﬁrms again that the reconstruction is the major cause of loss of variability.
However, we ﬁnd one interesting phenomenon regarding the quality of the samples.
The sample quality deteriorates up to a certain value of λ Rec , but gets back to normal as λ Rec further increases.
It implies that either
MLMM or the reconstruction loss can ﬁnd some high-quality local optima, but the joint use of them is not desirable.