C ONDITIONAL G ENERATIVE M ODELING VIA L EARNING THE L ATENT S PACE
Anonymous authors
Paper under double-blind review
A BSTRACT
Although deep learning has achieved appealing results on several machine learning tasks, most of the models are deterministic at inference, limiting their application to single-modal settings.
We propose a novel general-purpose framework for conditional generation in multimodal spaces, that uses latent variables to model generalizable learning patterns while minimizing a family of regression cost functions.
At inference, the latent variables are optimized to ﬁnd optimal solutions corresponding to multiple output modes.
Compared to existing generative solutions, our approach demonstrates faster and stable convergence, and can learn better representations for downstream tasks.
Importantly, it provides a simple generic model that can beat highly engineered pipelines tailored using domain expertise on a variety of tasks, while generating diverse outputs.
Our codes will be released.
I NTRODUCTION
Conditional generative models provide a natural mechanism to jointly learn a data distribution and optimize predictions.
In contrast, discriminative models improve predictions by modeling the label distribution.
Learning to model the data distribution allows generating novel samples and is considered a preferred way to understand the real world.
Existing conditional generative models have generally been explored in single-modal settings, where a one-to-one mapping between input and output domains exists (Nalisnick et al., 2019; Fetaya et al., 2020).
Here, we investigate continuous multimodal (CMM) spaces for generative modeling, where one-to-many mappings exist between input and output domains.
This is critical since many real world situations are inherently multi-modal, e.g., humans can imagine several outcomes for a given occluded image.
In a discrete setting, this problem becomes relatively easy to tackle using techniques such as maximum-likelihoodestimation, since the output can be predicted as a vector (Zhang et al., 2016), which is not possible in continuous domains.
One way to model CMM spaces is by using variational inference, e.g., variational autoencoders (VAE) (Kingma & Welling, 2013).
However, the approximated posterior distribution of VAEs are often restricted to the Gaussian family, which hinders its ability to model more complex distributions.
As a solution, Maaløe et al.
(2016) suggested using auxiliary variables to improve the variational distribution.
To this end, the latent variables are hierarchically correlated through injected auxiliary variables, which can produce non-Gaussian distributions.
A slightly similar work by Rezende & Mohamed (2015) proposed Normalizing Flows, that can hierarchically generate more complex probability distributions by applying a series of bijective mappings to an original simpler distribution.
Recently, Chang et al.
(2019) proposed a model, where a separate variable can be used to vary the impact of different loss components at inference, which allows diverse outputs.
For a more detailed discussion on these methods please see App. 1.
In addition to the aforesaid methods, in order to model CMM spaces, a prominent approach in the literature is to use a combination of reconstruction and adversarial losses (Isola et al., 2017;
Zhang et al., 2016; Pathak et al., 2016).
However, this entails key shortcomings.
The goals of adversarial and reconstruction losses are contradictory (Sec. 4), hence model engineering and numerous regularizers are required to support convergence (Lee et al., 2019; Mao et al., 2019), thereby resulting in less-generic models tailored for speciﬁc applications (Zeng et al., 2019; Vitoria et al., 2020).
The adversarial loss based models are notorious for difﬁcult convergence due to the challenge of ﬁnding Nash equilibrium of a non-convex min-max game in high-dimensions (Barnett, 2018; Chu et al., 2020; Kodali et al., 2017).
The convergence is heavily dependent on the architecture, hence such models show lack of scalability (Thanh-Tung et al., 2019; Arora &
Zhang, 2017).
The promise of assisting downstream tasks remains challenging, with a large gap in performance between the generative modelling approaches and their discriminative counterparts (Grathwohl et al., 2020; Jing & Tian, 2020).
In this work, we propose a general-purpose framework—Conditional Generation by Modeling the Latent Space (cGML)—for modeling CMM spaces using a set of domain-agnostic regression cost functions instead of the adversarial loss.
This improves both the stability and eliminates the incompatibility between the adversarial and reconstruction losses, allowing more precise outputs while maintaining diversity.
The underlying notion is to learn the ‘ behaviour of the latent variables ’ in minimizing these cost functions while converging to an optimum mode during the training phase, and mimicking the same at inference.
Despite being a novel direction, the proposed framework showcases promising attributes by: (a) achieving state-of-the-art results on a diverse set of tasks using a generic model, implying generalizability, (b) rapid convergence to optimal modes despite architectural changes, (c) learning useful features for downstream tasks, and (d) producing diverse outputs via traversal through multiple output modes at inference.
P ROPOSED M ETHODOLOGY
We deﬁne a family of cost functions { E i,j = d ( y gi,j , G ( x j , w )) } ∈ ξ , where x j ∼ χ is the input, y gi,j ∼ Υ is the i th ground-truth mode for x j , G is a generator function with weights w , and d ( · , · ) is a distance function.
Note that the number of cost functions E ( · ,j ) for a given x j can vary over χ .
Our aim here is to come up with a generator function G ( x j , w ) , that can minimize each E i,j , ∀ i as G ( x j , w ) → y gi,j .
However, since G is a deterministic function ( x and w are both ﬁxed at inference), it can only produce a single output.
Therefore, we introduce a latent vector z to the generator function, that can be used to converge ¯ y i,j = G ( x j , w, z i,j ) towards a y g ( i,j ) at inference, and possibly, to multiple solutions.
Formally, the family of cost functions now becomes: { ˆ E i,j = d ( y gi,j , G ( x j , w, z i,j )) } , ∀ z i,j ∼ ζ.
Then, our training objective can be deﬁned as ﬁnding a set of optimal z ∗ i ∈ ζ and w ∗ ∈ ω by minimizing E i ∼ I [ ˆ E i ] , where I is the number of possible solutions for x j .
Note that w ∗ is ﬁxed for all i and a different z ∗ i exists for each i .
Considering all the training samples x j ∼ χ , our training objective becomes,
[Equation]
Eq.
1 can be optimized via Algorithm 1 (proof in App. 2.2).
Intuitively, the goal of Eq.
1 is to obtain a family of optimal latent codes { z ∗ i,j } , each causing a global minima in the corresponding ˆ E i,j as y gi,j = G ( x j , w, z ∗ i,j ) .
Consequently, at inference, we can optimize ¯ y i,j to converge to an optimal mode in the output space by varying z .
Therefore, we predict an estimated ¯ z i,j at inference,
[Equation]
for each y gi,j , which in turn can be used to obtain the prediction G ( x j , ¯ z i,j , w )
≈ y gi,j .
In other words, for a selected x j , let ¯ y ti,j be the initial estimate for ¯ y i,j .
At inference, z can traverse gradually towards an optimum point y gi,j in the space, forcing ¯ y t + n i,j → y gi,j , in ﬁnite steps ( n ).
However, still a critical problem exists: Eq. 2 depends on y gi,j , which is not available at inference.
As a remedy, we enforce Lipschitz constraints on G over ( x j , z i,j ) , which bounds the gradient norm as,
[Equation]
where z 0 ∼ ζ is an arbitrary random initialization, C is a constant, and γ ( · ) is a straight path from z 0 to z ∗ i,j (proof in App. 2.1) .
Intuitively, Eq. 3 implies that the gradients ∇ z G ( x j , w ∗ , z 0 ) along the path γ ( · ) do not tend to vanish or explode, hence, ﬁnding the path to optimal z ∗ i,j in the space ζ becomes a fairly straight forward regression problem.
Moreover, enforcing the Lipschitz constraint encourages meaningful structuring of the latent space: suppose z ∗ 1 ,j and z ∗ 2 ,j are two optimal codes corresponding to two ground truth modes for a particular input.
Since (cid:107) z ∗ 2 ,j − z ∗ 1 ,j (cid:107) is lower bounded by (cid:107) G ( x j ,w ∗ ,z ∗ 2 ,j ) −G ( x j ,w ∗ ,z ∗ 1 ,j ) (cid:107) L , where L is the Lipschitz constant, the minimum distance between the two latent codes is proportional to the difference between the corresponding ground truth modes.
[Figure]
In practice, we observed that this encourages the optimum latent codes to be placed sparsely (visual illustration in App. 2), which helps a network to learn distinctive paths towards different modes.
2. C ONVERGENCE AT INFERENCE
We formulate ﬁnding the convergence path of z at inference as a regression problem, i.e., z t +1 = r
( z t , x j ) .
We implement r ( · ) as a recurrent neural network (RNN).
The series of predicted values { z ( t + k ) : k
= 1 , 2 , .., N } can be modeled as a ﬁrst-order Markov chain requiring no memory for the
RNN.
We observe that enforcing Lipschitz continuity on G over z leads to smooth trajectories even in high dimensional settings, hence, memorizing more than one step in to the history is redundant.
However, z t +1 is not a state variable, i.e., the existence of multiple modes for output prediction ¯ y leads to multiple possible solutions for z t +1 .
On the contrary, E [ z t +1 ] is a state variable w.r.t.
the state ( z t , x ) , which can be used as an approximation to reach the optimal z ∗ at inference.
Therefore, instead of directly learning r ( · ) , we learn a simpliﬁed version r (cid:48)
( z t , x ) =
E [ z t +1 ] .
Intuitively, the whole process can be understood as observing the behavior of z on a smooth surface at the training stage, and predicting the movement at inference.
A key aspect of r (cid:48)
( z t , x ) is that the model is capable of converging to multiple possible optimum modes at inference based on the initial position of z .
2. M OMENTUM AS A SUPPLEMENTARY AID
Based on Sec. 2.1, z can now traverse to an optimal position z ∗ during inference.
However, there can exist rare symmetrical positions in the ζ where E [ z t +1 ] − z t ≈ 0 , although far away from { z ∗ } , forcing z t +1 ≈ z t .
Simply, the above phenomenon can occur if some z t +1 has traveled in many non-orthogonal directions, so the vector addition of z t +1 ≈ 0 .
This can fool the system to falsely identify convergence points, forming phantom optimum point distributions amongst the true distribution (see Fig. 3).
To avoid such behavior, we consider (cid:126)v ( z t , x j ) =
( z t +1 − z t ) x j .
Then, we learn the expected momentum E [ ρ ( z t , x j )]
= α E [ | (cid:126)v ( z t , x j ) | ] at each ( z t , x j ) during the training phase, where α is an empirically chosen scalar.
In practice, E [ ρ ( z t , x j )] → 0 as z t +1 , z t → { z ∗ } .
Thus, to avoid phantom distributions, we improve the z update as,
[Equation]
Since both E [ ρ ( z t , x j )] and r (cid:48) ( z t , x j ) are functions on ( z t , x j ) , we jointly learn these two functions using a single network Z ( z t , x j ) .
Note that coefﬁcient E [ ρ ( z t , x j )] serves two practical purposes: 1) slows down the movement of z near true distributions, 2) pushes z out of the phantom distributions.
O VERALL D ESIGN
The proposed model consists of three major blocks as shown in Fig.
1: an encoder H , a generator G , and Z .
The detailed architecture diagram for 128 × 128 is shown in Fig.
Note that for derivations in
Sec. 2, we used x instead of h = H ( x ) , as h is a high-level representation of x .
The training process is illustrated in Algorithm 1.
At each optimization z t +1 =
z t − β ∇ z t [ ˆ E i,j ] , Z is trained separately to approximate ( z t +1 , ρ ) .
At inference, x is fed to H , and then the Z optimizes the output ¯ y by updating z for a pre-deﬁned number of iterations of Eq. 4.
For ˆ E ( · , · ) , we use L 1 loss.
Furthermore, it is important to limit the search space for z t +1 , to improve the performance of Z .
To this end, we sample z from the surface of the n -dimensional sphere ( S n ).
Moreover, to ensure faster convergence of the model, we force the Lipschitz continuity on both Z and the G (App. 2.4) .
For hyper-parameters and training details, see App. 3.1.
sample inputs { x 1 , x 2 , ..., x J } ∈ χ ; sample outputs { y 1 , y 2 , ..., y J } ∈ Υ ; for k epochs do for x in χ do for l steps do update z = { z 1 , z 2 , ..., z J } : ∇ z ˆ E (cid:66) Freeze H , G , Z and update z update Z : ∇ w L 1 [( z t +1 , ρ ) , Z ( z t , H ( x ))] (cid:66) Freeze H , G , z and update Z update G , H : ∇ w ˆ E (cid:66)
Freeze Z , z and update H , G
M OTIVATION
Here, we explain the drawbacks of conditional GAN methods and illustrate our idea via a toy example.
Incompatibility of adversarial and reconstruction losses: cGANs use a combination of adversarial and reconstruction losses.
We note that this combination is suboptimal to model CMM spaces.
Remark: Consider a generator G ( x, z ) and a discriminator D ( x, z ) , where x and z are the input and the noise vector, respectively.
Then, consider an arbitrary input x j and the corresponding set of ground-truths { y gi,j } , i = 1 , 2 , ..N .
Further, let us deﬁne the optimal generator G ∗ ( x j , z )
=
ˆ y, ˆ y ∈ { y gi,j } , L GAN = E i [log D ( y gi,j )]
+ E z
[log(1 − D ( G ( x j , z ))] and L (cid:96) =
E i,z [ | y gi,j − G ( x j , z ) | ] .
Then,
G ∗ (cid:54) =
ˆ G ∗ where ˆ G ∗ =
arg G min D max L GAN + λL (cid:96) , ∀ λ (cid:54)
= 0 .
(Proof in App. 2.3).
Generalizability: The incompatibility of above mentioned loss functions demands domain speciﬁc design choices from models that target high realism in CMM settings.
This hinders the generalizability across different tasks (Vitoria et al., 2020; Zeng et al., 2019).
We further argue that due to this discrepancy, cGANs learn sub-optimal features which are less useful for downstream tasks (Sec. 5.3).
Convergence and the sensitivity to the architecture:
The difﬁculty of converging GANs to the
Nash equilibrium of a non-convex min-max game in high-dimensional spaces is well explored (Barnett, 2018;
Chu et al., 2020; Kodali et al., 2017).
Goodfellow et al.
(2014b) underlines if the discriminator has enough capacity, and is optimal at every step of the GAN algorithm, then the generated distribution converges to the real distribution ; that cannot be guaranteed in a practical scenario.
In fact, Arora et al.
conﬁrmed that the adversarial objective can easily approach to an equilibrium even if the generated distribution has very low support, and further, the number of training samples required to avoid mode collapse can be in order of exp( d ) ( d is the data dimension).
Multimodality: The ability to generate diverse outputs, i.e., convergence to multiple modes in the output space, is an important requirement.
Despite the typical noise input, cGANs generally lack the ability to generate diverse outputs (Lee et al., 2019).
Pathak et al.
(2016) and Iizuka et al.
(2016) even state that better results are obtained when the noise is completely removed.
Further, variants of cGAN that target diversity often face a trafe-off between the realism and diversity (He et al., 2018), as they have to compromise between the reconstruction and adversarial losses.
[Figure]
A toy example: Here, we experiment with the formulations in Sec. 2.
Consider a 3D CMM space y = ±
4( x, x 2 , x 3 ) .
Then, we construct three layer multi-layer perceptrons (MLP) to represent each of the functions, H , G , and Z , and compare the proposed method against the L 1 loss.
Figureillustrates the results.
As expected, L 1 loss generates the line y = 0 , and is inadequate to model the multimodal space.
As explained in Sec. 2.2, without momentum correction, the network is fooled by a phantom distribution where E [ z t +1 ]
≈ 0 at training time.
However, the push of momentum removes the phantom distribution and reﬁnes the output to closely resemble the input distribution.
As implied in Sec.
2.2, the momentum is maximized near the true distribution and minimized otherwise.
E XPERIMENTS AND DISCUSSIONS
The distribution of natural images lies on a high dimensional manifold, making the task of modelling it extremely challenging.
Moreover, conditional image generation poses an additional challenge with their constrained multimodal output space (a single input may correspond to multiple outputs while not all of them are available for training).
In this section, we experiment on several such tasks.
For a fair comparison with a similar capacity GAN, we use the encoder and decoder architectures used in
Pathak et al.
(2016) for H and G respectively.
We make two minor modiﬁcations: the channel-wise fully connected (FC) layers are removed and U-Net style skip connections are added (see App. 3.1).
We train the existing models for a maximum of 200 epochs where pretrained weights are not provided, and demonstrate the generalizability of our theoretical framework in diverse practical settings by using a generic network for all the experiments.
Models used for comparisons are denoted as follows:
PN (Zeng et al., 2019), CA (Yu et al., 2018b), DSGAN (Yang et al., 2019), CIC (Zhang et al., 2016),
RFR (Li et al., 2020), Chroma (Vitoria et al., 2020), P2P (Isola et al., 2017), Izuka (Iizuka et al., 2016), CE (Pathak et al., 2016), CRN (Chen & Koltun, 2017a), and B-GAN (Zhu et al., 2017b).
5. C ORRUPTED I MAGE R ECOVERY
We design this task as image completion, i.e., given a masked image as input, our goal is to recover the masked area.
Interestingly, we observed that the MNIST dataset, in its original form, does not have a multimodal behaviour, i.e., a fraction of the input image only maps to a single output.
Therefore, we modify the training data as follows:
ﬁrst, we overlap the top half of an input image with the top half of another randomly sampled image.
We carry out this corruption for 20% of the training data.
Corrupted samples are not ﬁxed across epochs.
Then, we apply a random sized mask to the top half, and ask the network to predict the missing pixels.
We choose two competitive baselines here: our network with the L 1 loss and CE.
Fig.
4 illustrates the predictions.
As shown, our model converges to the most probable non-corrupted mode without any ambiguity, while other baselines give sub-optimal results.
In the next experiment, we add a small white box to the top part of the ground-truth images at different rates.
At inference, our model was able to converge to both the modes (Fig. 5), depending on the initial position of z , as the probability of the alternate mode reaches 0 .
5. A UTOMATIC IMAGE COLORIZATION
Deep models have tackled this problem using semantic priors (Iizuka et al., 2016; Vitoria et al., 2020), adversarial and L 1 losses (Isola et al., 2017; Zhu et al., 2017a; Lee et al., 2019), or by conversion to a discrete form through binning of color values (Zhang et al., 2016).
Although these methods provide
[Table] [Table]
[Table]
compelling results, several inherent limitations exist: (a) use of semantic priors results in complex models, (b) adversarial loss suffers from drawbacks (see Sec. 4), and (c) discretization reduces the precision.
In contrast, we achieve better results using a simpler model.
The input and the output of the network are l and ( a, b ) planes respectively (LAB color space).
However, since the color distributions of a and b spaces are highly imbalanced over a natural dataset (Zhang et al., 2016)
[Table]
we add another constraint to the cost function E to push the predicted a and b colors towards a uniform distribution: E = (cid:107) a gt − a (cid:107) + (cid:107) b gt − b (cid:107) + λ ( loss kl,a + loss kl,b ) , where loss kl, · = KL( ·|| u (0 , 1)) .
Here, KL( ·||· ) is the KL divergence and u (0 , 1) is a uniform distribution (see App. 3.3).
Fig.
7 and Table 2 depict our qualitative and quantitative results, respectively.
We demonstrate the superior performance of our method against four metrics: LPIP, PieAPP, SSIM and
PSNR (App. 3.2).
Fig.
10 depicts examples of multimodality captured by our model (more examples in App. 3.4).
Fig.
6 shows colorization behaviour as the z converges during inference.
User study: We also conduct two user studies to further validate the quality of generated samples (Table 1).
a)
In the P SYCHOPHYSICAL STUDY , we present volunteers with batches of 3 images, each generated with a different method.
A batch is displayed for 5 secs and the user has to pick the most realistic image.
After 5 secs, the next image batch is displayed.
b) We conduct a T URING TEST to validate our output quality against the ground-truth, following the setting proposed by Zhang et al.
The volunteers are presented with a series of paired images (ground-truth and our output).
The images are visible for 1 sec, and then the user has an unlimited time to pick the realistic image.
5. I MAGE COMPLETION
In this case, we show that our generic model outperforms a similar capacity GAN (CE) as well as task-speciﬁc GANs.
In contrast to task-speciﬁc models, we do not use any domain-speciﬁc modiﬁcations to make our outputs perceptually pleasing.
We observe that with random irregular and ﬁxed-sized masks, all the models perform well, and we were not able to visually observe a considerable difference (Fig. 8, see App. 3.11 for more results).
Therefore, we presented models with a more challenging task: train with random sized square-shaped masks and evaluate the performance t
L
C E
O u r s
[Figure] [Figure]
G T
I z uk a
P 2 P
C h r o m a
C I C
O u r s
G T
I npu t
O u t pu t
[Figure]
PN
RFR [Figure]
[Figure]
against masks of varying sizes.
Fig.
9 illustrates qualitative results of the models with 25% masked data.
As evident, our model recovers details more accurately compared to the state-of-the-art.
Notably, all models produce comparable results when trained with a ﬁxed sized center mask, but ﬁnd this setting more challenging.
Table 3 includes a quantitative comparison.
Observe that in the case of smaller sized masks, PN performs slightly better than ours, but worse otherwise.
We also evaluate the learned features of the models against a downstream classiﬁcation task (Table 5).
First, we train all the models on Facades (Tyle ˇ cek & Šára, 2013) against random masks, and then apply the trained models on CIFAR10 (Krizhevsky et al., 2009) to extract bottleneck features, and ﬁnally pass them through a FC layer for classiﬁcation (App. 3.7).
We compare PN and ours against an oracle (AlexNet features pre-trained on ImageNet) and show our model performs closer to the oracle.
5.3. D IVERSITY AND OTHER COMPELLING ATTRIBUTES
We also experiment on a diverse set of image translation tasks to demonstrate our generalizability.
Fig.
12, 13, 14, 16 and 17 illustrate the qualitative results of sketch-to-handbag , sketch-to-shoes , map-to-arial , lanmarks-to-faces and surface-normals-to-pets tasks.
Fig.
10, 11, 12, 13, 16 andshow the ability of our model to converge to multiple modes, depending on the z initialization.
Fig.demonstrates the quantitative comparison against other models.
See App. 3.4 for further details on experiments.
Another appealing feature of our model is its strong convergence properties irrespective of the architecture, hence, scalability to different input sizes.
Fig.
19 shows examples from image completion and colorization for varying input sizes.
We add layers to the architecture to be trained
Our model exhibits rapid and stable convergence compared to state-of-the-art (PN, CE, P2P, CA).
[Table] [Table]
[Table] [Table]
on increasingly high-resolution inputs, where our model was able to converge to optimal modes at each scale (App. 3.8).
Fig.
18 demonstrates our faster and stable convergence .
Table.
7 compares the number of ﬂops required by the models for a batch size of 10 .
5. D ENOISING OF 3D OBJECTS IN SPECTRAL SPACE
Spectral moments of 3D objects provide a compact representation, and help building light-weight networks (Ramasinghe et al., 2020; 2019b;
Cohen et al., 2018; Esteves et al., 2018).
However, spectral information of 3D objects has not been used before for self-supervised learning, a key reason being the difﬁculty of learning representations in the spectral domain due to the complex structure and unbounded spectral coefﬁcients.
Here, we present an efﬁcient pretext task that is conducted in the spectral domain: denoising 3D spectral maps.
We use two types of spectral spaces: spherical harmonics and Zernike polynomials (App. 4).
We ﬁrst convert the 3D point clouds to spherical harmonic coefﬁcients, arrange the values as a 2D map, and mask or add noise to a map portion (App. 3.12).
The goal is to recover the original spectral map.
Fig.
20 and Table 6 depicts our qualitative and quantitative results.
We perform favorably well against other methods.
To evaluate the learned features, we use Zernike polynomials, as they are more discriminative compared to spherical harmonics (Ramasinghe et al., 2019a).
We ﬁrst train the network on the 55k ShapeNet objects by denoising spectral maps, and then apply the trained network on the ModelNet10 & 40.
The features are then extracted from the bottleneck (similar to Sec. 5.3), and fed to a FC classiﬁer (Table 4).
We achieve the state-of-the-art results in ModelNet40 with a simple pretext task.
[Figure] [Figure]
C ONCLUSION
Conditional generation in multimodal domains is a challenging task due to its ill-posed nature.
In this paper, we propose a novel generative framework that minimize a family of cost functions during training.
Further, it observes the convergence patterns of latent variables and applies this knowledge during inference to traverse to multiple output modes during inference.
Despite using a simple and generic architecture, we show impressive results on a diverse set of tasks.
The proposed approach demonstrates faster convergence, scalability, generalizability, diversity and superior representation learning capability for downstream tasks.
A PPENDIX R ELATED WORK
Conditional Generative Modeling.
Conditional generation involves modeling the data distribution given a set of conditioning variables that control of modes of the generated samples.
With the success of VAEs (Kingma & Welling, 2014) and GANs (Goodfellow et al., 2014a) in standard generative modeling tasks, their conditioned counterparts (Sohn et al., 2015; Mirza & Osindero, 2014) have dominated conditional generative tasks recently (Vitoria et al., 2020; Zhang et al., 2016; Isola et al., 2017; Pathak et al., 2016; Lee et al., 2019; Zhu et al., 2017a; Bao et al., 2017; Lee et al., 2018;
Zeng et al., 2019).
While probabilistic latent variable models such as VAEs generate relatively low quality samples and poor likelihood estimates at inference (Maaløe et al., 2019), GAN based models perform signiﬁcantly better at high dimensional distributions like natural images but demonstrate unstable training behaviour.
A distinct feature of GANs is its mapping of points from a random noise distribution to the various modes of the output distribution.
However, in the conditional case where an additional loss is incorporated to enforce the conditioning on the input, the signiﬁcantly better performance of GANs is achieved at the expense of multimodality; the conditioning loss pushes the
GAN to learn to mostly ignore its noise distribution.
In fact, some works intentionally ignore the noise input in order to achieve more stable training (Isola et al., 2017; Pathak et al., 2016; Mathieu et al., 2015; Xie et al., 2018).
Multimodality.
Conditional VAE-GANs are one popular approach for generating multimodal outputs (Bao et al., 2017; Zhu et al., 2017a) using the VAE’s ability to enforce diversity through its latent variable representation and the GAN’s ability to enforce output ﬁdelity through its learnt discrimanator model.
Mixture models (Chen & Koltun, 2017b; Ghosh et al., 2018; Deshpande et al., 2017) that discretize the output space are another approach.
Domain speciﬁc disentangled representations (Lee et al., 2018; Huang et al., 2018) and explicit encoding of multiple modes as inputs Zhu et al.
Isola et al.
(2017) have also been successful in generating diverse outputs.
Sampling-based loss functions enforcing similarity at a distribution level (Lee et al., 2019) have also been successful in multimoal generative tasks.
Further, the use of additional specialized reconstruction losses (often using higher-level features extracted from the data distribution) and attention mechanisms also achieves multimodality through intricate model architectures in domain speciﬁc cases (Zeng et al., 2019; Chen & Koltun, 2017b; Vitoria et al., 2020; Zhang et al., 2016; Iizuka et al., 2016; Zhang et al., 2020; Yu et al., 2018a; Sagong et al., 2019; Wang et al., 2018;
Iizuka et al., 2016).
We propose a simpler direction through our domain-independent energy function based approach that is also capable of learning generic representations that better support downstream tasks.
Notably, our work contrasts from energy based models previously investigated for likelihood modeling due to their simplicity, however, such models are notoriously difﬁcult to train especially on high-dimensional spaces (Du & Mordatch, 2019).
VAEs and other related work.
Variational auto encoders (VAE) are a special class of autoencoders that are trained in a manner that ensures the latent space has good properties allowing the generation of new data.
In VAEs, for an input x , the latent space is modeled as a probability distribution q ( z | x ) , which is sampled from a known family of distributions (typically Gaussian), as the true posterior distribution p ( z | x ) is intractable.
However, assuming the posterior distribution of the latent space as a
Gaussian distribution constrains the quality of the generated data distribution, as the true distribution may be far from a Gaussian.
Therefore, it is beneﬁcial to model q ( z | x ) as a more complex distribution, in order to generate high dimensional data distributions.
As a solution, Maaløe et al.
(2016) suggested using a set of auxiliary variables a to improve the ﬂexibility of q ( z | x ) .
The key idea is to obtain a complex marginal q ( z | x ) = (cid:82) q
( z | a, x ) p ( a, x ) da , which can be non-Gaussian.
On the other hand, Normalizing ﬂows (NF) (Rezende & Mohamed, 2015), among other beneﬁts, provides an ideal mechanism for the above task.
NFs apply a series of bijective mappings NF : P ( z i )
→ P ( z i +1 ) , where P ( z i +1 ) is typically more complex compared to
P ( z i ) .
In contrast, we do not explicitly model our latent space as a probability distribution.
However, we draw some interesting analogies from a probabilistic perspective as follows: our latent space ζ can be interpreted as a set of energy surfaces E x j : ζ → R , as E x j = ||
y gj − G ( x, z j ) || for each ground truth mode y gj .
From this perspective, Fig.
32 in the appendix illustrates the energy heatmaps for the toy example.
As shown, high energies are indicated by a brighter color.
Since our system has a ﬁnite energy, the combined energy E x = (cid:80) j E x j can be transformed to a probability distribution via the
Gibbs measure as p (cid:48)
( z ) = 1 T ( β ) exp( − βE x
( z )) , where T ( · ) is the partition function.
Note that this probability is not restricted to a simple distribution.
A critical difference between the VAEs and our model is that we do not sample directly from p (cid:48)
( z ) , since to obtain p (cid:48) ( z ) , we need to integrate E x over the latent space.
However, our predictor network
Z learns the high probability coordinates { z ∗ } of p (cid:48)
( z ) , and is able to converge to such locations at inference.
This probabilistic perspective of our latent space (or the corresponding energy surface) is intuitively justiﬁed by the convergence samples shown in Fig.
52 in appendix.
The intermediate samples we obtain as we go from z to z ∗ also produce plausible results, however, the visual quality at the z ∗ is maximized, indicating high p (cid:48)
( z = z ∗ | x ) .
In contrast to NFs, our model does not explicitly learn the probability distributions, rather, the predictor network learns to converge to the high probability areas in complex distributions.
Also, the complexity of the p (cid:48)
( z ) increases with the complexity and the multimodality of the corresponding higher dimensional target data distribution.
Therefore, the required dimensionality of the z tends to increase in such cases.
A property of NFs is that each transformation affects only a small volume in the original space, hence, we need a higher number of layers to work with high dimensional spaces (the volume grows exponentially with the dimension of the space).
In contrast, we did not observe such an increase in the required capacity of the predictor with the dimension of z .
The model proposed by Chang et al.
(2019) also uses a separate input S , that can vary the output of the generator, in cases where multiple loss components are used.
The variable S is used both as an input to the generator and also to control the weights of the loss components while training.
Interestingly, at the inference, the model is able to approximate the change in loss based on the input
S , and generate diverse outputs.
However, this method is more useful in scenarios where the required diversity is in the form of different styles, which can be induced by different loss functions.
T HEORETICAL RESULTS 2. P ROOF FOR E Q .
[Equation]
Let γ ( t ) be a straight path from z 0 to z ∗ i,j , where γ (0) =
z 0 and γ (1) =
z ∗ i,j .
Then,
[Equation] [Equation]
On the other hand the Lipschitz constraint ensures,
[Equation]
where C is a constant.
Combining Eq.
9 and 10 we get,
[Equation]
2. C ONVERGENCE OF THE TRAINING ALGORITHM .
Proof: Let us consider a particular input x j and an associated ground truth y gi,j .
Then, for this particular case, we denote our cost function to be ˆ E i,j = d ( w, z ) .
Further, a family of cost functions can be deﬁned as,
[Equation]
for each w ∼ ω .
Further, let us consider an arbitrary initial setting ( z init , w init ).
Then, with enough iterations, gradient descent by ∇ z f w ( z ) converges z init to,
[Equation]
Next, with enough iterations, gradient descent by ∇ w f w (¯ z ) converges w to,
[Equation]
Observe that f ¯ w (¯ z ) ≤ f w init , where the equality occurs when ∇ z f w ( z ) = ∇ w f w (¯ z ) = 0 .
If f w ( z ) has a unique global minima, repeating Equation 13 and 14 converges to that global minima, giving { z ∗ i,j , w ∗ i,j } .
It is straight forward to see that using a small number of iterations (usually one in our case) for each sample set for Equation 14, i.e., stochastic gradient descent, gives us,
[Equation]
) where w ∗ is ﬁxed for all samples and modes (Robbins, 2007).
Note that the proof is valid only for the convex case, and we rely on stochastic gradient descent to converge to at least a good local minima, as commonly done in many deep learning settings.
2. P ROOF FOR R EMARK
Remark: Consider a generator G ( x, z ) and a discriminator D ( x, z ) with a ﬁnite capacity, where x and z are input and the noise vector, respectively.
Then, consider an arbitrary input x j and the corresponding set of ground truths { y gi,j } , i = 1 , 2 , ..N .
Further, let us deﬁne the optimal generator G ∗ ( x j , z )
=
ˆ y, ˆ y ∈ { y gi,j } , L GAN = E i [log D ( y gi,j )]
+ E z
[log(1 − D ( G ( x j , z ))] and
L (cid:96) =
E i,z [ | y gi,j − G ( x j , z ) | ] .
Then, G ∗ (cid:54) =
ˆ G ∗ where ˆ G ∗ =
arg G min D max L GAN + λL (cid:96) , ∀ λ (cid:54)
= 0 .
Proof.
It is straightforward to derive the equilibrium point of arg G min D max L GAN from the original GAN formulation.
However, for clarity, we show some steps here.
Let,
[Equation] [Equation]
Let p ( · ) denote the probability distribution.
Then,
[Equation]
Consider the inner loop.
It is straightforward to see that V ( G, D ) is maximized w.r.t.
D when
D
[Equation]
Then,
[Equation]
Then, following the Theorem 1 from Goodfellow et al.
(2014b)
, it can be shown that the global minimum of the virtual training criterion C ( G ) is achieved if and only if p ( y g · ,j ) =
p (¯ y · ,j ) .
Next, consider the L 1 loss for x j ,
[Equation] [Equation]
[Figure]
[Equation]
For L 1 to approach to a minima, ∇ w L 1 → 0 .
Since { y gi,j } is not a singleton, when L 1 → 0 ,
G ( x j , z, w ) (cid:54) =
ˆ y ∈ { y gi,j } .
Now, let us consider the L 2 loss,
[Equation]
For ∇ w L 2 → 0 , G ( x j , z, w ) → 1 N (cid:80) i y gi,j .
However, omitting the very speciﬁc case where ( 1 N (cid:80) i
y gi,j ) ∈ { y gi,j } , which is highly unlikely in a complex distribution, as L 2 → 0 , G ( x j , z, w ) (cid:54) =
ˆ y ∈ { y gi,j } .
Therefore, the goals of arg G min D max L GAN and λL (cid:96) are contradictory and G ∗ (cid:54)
=
ˆ G ∗ .
Note that we do not extend our proof to high order L losses as it is intuitive.
2. L IPSCHITZ CONTINUITY AND STRUCTURING OF THE LATENT SPACE
Enforcing the Lipschitz constraint encourages meaningful structuring of the latent space: suppose z ∗ 1 ,j and z ∗ 2 ,j are two optimal codes corresponding to two ground truth modes for a particular input.
Since (cid:13)(cid:13)(cid:13) z ∗ 2 ,j − z ∗ 1 ,j (cid:13)(cid:13)(cid:13) is lower bounded by (cid:107) G ( x j ,w ∗ ,z ∗ 2 ,j ) −G ( x j ,w ∗ ,z ∗ 1 ,j ) (cid:107) L , where L is the Lipschitz constant, the minimum distance between the two latent codes is proportional to the difference between the corresponding ground truth modes.
Also, in practice, we observed that this encourages the optimum latent codes to be placed sparsely.
Fig.
21 illustrates a visualization from the toy example.
As the training progresses, the optimal { z ∗ } corresponding to minimas of ˆ E are identiﬁed and placed sparsely.
Note that as expected, at the 10 th epoch the distance between the two optimum z ∗ increases as x goes from 0 to 1 , in other words, as the (cid:13)(cid:13) 4( x, x 2 , x 3 ) − ( − 4( x, x 2 , x 3 ))
(cid:13)(cid:13) increases.
Practical implementation is done as follows: during the training phase, a small noise e is injected to the inputs of Z and G , and the networks are penalized for any difference in output.
More formally,
L Z and ˆ E now become, L 1 [ z t +1 , Z
( z t , h )]
+ αL 1 [ Z ( z t + e, h + e ) , Z ( z t , h )] and L 1 [ y g , G ( h, z )] + αL 1 [ G ( h + e, z + e ) , G ( h, z )] , respectively.
Fig.
25 illustrates the procedure.
2. T OWARDS A MEASUREMENT OF UNCERTAINTY
In Bayesian approaches, the uncertainty is represented using the distribution of the network parameters ω .
Since a network output is unique for ﬁxed ¯ w ∼ ω , sampling from the output is equivalent to sampling from ω .
Often, ω is modeled as a parametric distribution or obtained through sampling, and at inference, the model uncertainty can be estimated as VAR p ( y | x ) ( y ) .
One intuition behind this is that for more conﬁdent inputs, p ( y | x, w ) will showcase less variance over the distribution of ω —hence lower VAR p ( y | x ) ( y ) —as the network parameters have learned redundant information (Loquercio et al., 2019).
As opposed to sampling from the distribution of network parameters, we model the optimal z ∗ for a particular input as a probability distribution p ( z ∗ ) , and measure VAR p ( y | x ) ( y ) where p ( y | x ) =
(cid:82) p ( y | x, z ∗ ) p ( z ∗ | x ) dz .
Our intuition is that in the vicinity of well observed data VAR p ( y | x ) ( y ) is lower, since for training data 1) we enforce the Lipschitz constraint on G ( x, z ) over ( x, z ) and 2) ˆ E
( y g , G ( x, z )) resides in a relatively stable local minima against z ∗ for observed data, as in practice, z ∗ =
E epochs [ z ∗ ] + (cid:15) for a given x , where (cid:15) is some random noise which is susceptible to change over each epoch.
Further, Let ( x, z ∗ ) and y g be the inputs to a network G and the corresponding ground truth label, respectively.
Formally, let p ( y g | x, z ∗ ) =
N ( y g ; G ( x, z ∗ ) , α I ) and z ∗ ∼ U ( | z ∗ − E ( z ∗ ) | < δ ) , where α is some variable describing the noise in the input x and δ is a small positive scalar.
Then,
[Equation]
where COV is the sample covariance.
proof:
[Equation]
Let πδ
=
A
[Equation]
and p ( z ∗ | x ) ≈ 1 A .
Then
[Equation]
by Monte-Carlo approximation
[Equation]
Next
[Equation]
consider,
[Equation]
Note that in similar to Bayesian uncertainty estimations, where an approximate distribution q ( w ) is used to estimate p ( w | D ) , where D is data, our model sample from the an empirical distribution p ( z ∗ | x ) .
In practice, we treat α k as a constant over all the samples–hence omit from the calculation— and use stochastic forward passes to obtain Eq.
Then, the diagonal entries are used to calculate the uncertainty in the each dimension of the output.
We test this hypothesis on the toy example and the colorization task, as shown in Fig.
22 and Fig. 23, respectively.
E XPERIMENTS 3. E XPERIMENTAL ARCHITECTURES
For the experiments on images, we mainly use 128 × 128 size inputs.
However, to demonstrate the scalability, we use several different architectures and show that the proposed framework is capable of converging irrespective of the architecture.
Fig.
24 shows the architectures for different input sizes.
For training, we use the Adam optimizer with hyper-parameters β 1 = 0 .
9 , β 2 = 0 .
999 , (cid:15) = 1 × 10 − 8 , and a learning rate lr = 1 × 10 − 5 .
We use batch normalization after each convolution layer, and
[Table]
leaky ReLu as the activation, except the last layer where we use tanh .
All the weights are initialized using a random normal distribution with 0 mean and 0 .
5 standard deviation.
Furthermore, we use a batch size of 20 for training, though we did not observe much change in performance for different batch sizes.
We choose the dimensions of z to be 10 , 16 , 32 , 64 for 32 × 32 , 64 × 64 , 128 × 128 , 256 × 256 input sizes, respectively.
An important aspect to note here is that the dimension of z should not be increased too much, as it would increase the search space for z unnecessarily.
While training, z is updated 20 times for a single G , H update.
Similarly, at inference, we use 20 update steps for z , in order to converge to the optimal solution.
All the values are chosen empirically.
3. E VALUATION METRICS
Although heavily used in the literature, per pixel metrics such as PSNR do es not effectively capture the perceptual quality of an image.
To overcome this shortcoming, more perceptually motivated metrics have been proposed such as SSIM Wang et al.
(2004), MSSIM Wang et al.
(2003), and FSIM
Zhang et al.
However the similarity of two images is largely context dependant, and may not be captured by the aforementioned metrics.
As a solution, recently, two deep feature based perceptual metrics–LPIP Zhang et al.
(2018) and PieAPP Prashnani et al.
(2018)–were proposed, which coincide well with the human judgement.
To cover all these aspects, we evaluate our experiments against four metrics: LPIP, PieAPP, PSNR and SSIM.
3. U NBALANCED COLOR DISTRIBUTIONS
The color distribution of a natural dataset in a and b planes (LAB space) are strongly biased towards low values.
If not taken into account, the loss function can be dominated by these desaturated values.
Richard et al.
Zhang et al.
(2016) addressed this problem by rebalancing class weights according to the probability of color occurrence.
However, this is only possible in a case where the output domain is discretized.
To tackle this problem in the continuous domain, we push the output color distribution towards a uniform distribution as explained in Sec.
5.2 in the main paper.
3. M ULTIMODALITY
An appealing attribute of our network is its ability to converge to multiple optimal modes at inference.
A few such examples are shown in Fig.
27, Fig.
26, Fig. 31 Fig. 32, Fig. 29, Fig. 30 and Fig.
For the facial-land-marks-to-faces experiment, we used the UTKFace dataset (Zhang & Qi, 2017).
For the surface-normals-to-pets experiment, we used the Oxford Pet dataset (Parkhi et al., 2012).
In order to get the surface normal images, we follow Bansal Bansal et al.
(2017b).
First, we crop the bounding boxes of pet faces and then apply PixelNet (Bansal et al., 2017a) to extract surface normals.
For maps-to-ariel and edges-to-photos experiments, we used the datasets provided by Isola et al.
For measuring the diversity, we adapt the following procedure: 1) we generate 20 random samples from the model.
2) calculate the mean pixel value µ i of each sample.
3) pick the closest sample s m to the average of all the mean pixels λ = 120 (cid:80)
i =1 µ i . 4) pick the 10 samples which have maximum mean pixel distance from s m .
5) calculate the mean standard deviation of the 10 samples picked in step 4. 6) repeat the experiment 5 times for each model and get the expected standard deviation.
3. C OLORIZATION ON STL DATASET
Additional colorization examples on the STL dataset are shown in Fig.
We also compare the color distributions of the predicted a, b planes with state-of-the-art.
The results are shown in Fig.
33 and
Table 8.
As evident, our method predicts the closest color distribution to the ground truth.
3. C OLORIZATION ON I MAGE N ET DATASET
Additional colorization examples on the ImageNet dataset are shown in Fig.
[Table]
3. S ELF - SUPERVISED LEARNING SETUP
Here we evaluate the performance of our model on down-stream tasks, using three distinct setups involving bottleneck features of trained models.
The bottleneck layer features (of models trained on some dataset) are fed to a fully-connected layer and trained on a different dataset.
The baseline experiment uses the output of the penultimate layer in a Resnet-50 trained on ImageNet for classiﬁcation as the bottleneck features.
The comparison to state-of-the-art experiment involves
Zeng et al.
(2019) where the ﬁve outputs of its multi-scale decoder are max-pooled and concatenated to use as the bottleneck features.
The outputs of layers before this were also experimented with, and the highest performance was obtained for these selected features.
In our network, the output of the encoder network was used as the bottleneck features.
3. S CALABILITY
One promising attribute of the proposed method compared to the state-of-the-art is its scalability.
In other words, we propose a generic framework which is not bound to the architecture, hence, the model can be scaled to different input sizes without affecting the convergence behaviour.
To demonstrate this, we use 4 different architectures and train them on 4 different input sizes ( 32 × 32 , 64 × 64 , 128 × 128 , 256 × 256 ) on the same tasks: image completion and colorization.
The different architectures we use are shown in Fig.
3. A BLATION STUDY ON THE z DIMENSION
To demonstrate the effect of dimension of z on the model accuracy, we conduct an ablation study for the colorization task for the input size 128 × 128 .
Table 9 shows the results.
The quality of the outputs increases to a maximum when dim ( z ) = 32 , and then decreases.
This is intuitive because when the search space of z gets unnecessarily high, it becomes difﬁcult for Z to learn the paths to optimum modes, due to limited capacity.
3. U SER STUDIES
Evaluation of synthesized images is an open problem (Salimans et al., 2016).
Although recent metrics such as LPIP (Zhang et al., 2018) and PieAPP (Prashnani et al., 2018) have been proposed,
[Table]
which coincide closely with human judgement, perceptual user studies remain the preferred method.
Therefore, to evaluate the quality of our synthesized images in the colorization task, we conduct two types of user studies: a Turing test and a psychophysical study.
In the Turing test, we show the users a series of paired images, ground truth and our predictions, and ask the users to pick the most realistic image.
Here, following Zhang et al.
, we display each image for 1 second, and then give the users an unlimited amount of time to make the choice.
For the psychophysical study, we choose the two best performing methods according to the LPIP metric: Vitoria et al.
(2020) and Iizuka et al.
We create a series of batches of three images, Vitoria et al.
(2020), Iizuka et al.
(2016) and ours, and ask the users to pick the best quality image.
In this case, each batch is shown to the users for 5 seconds, and the users have to make this decision during that time.
We conduct the Turing test on ImageNet, and the psychophysical study on both ImageNet and STL datasets.
For each test, we use 500 randomly sampled batches and ∼ 15 users.
We also conduct Turing tests to evaluate the image completion tasks on Facades and Celeb-HQ datasets.
The results are shown in Table 10.
3. I MAGE COMPLETION
The additional image completion examples are provided in Figs.
36 and 37.
Our turing test results on
Celeb-HQ and Facades are shown in Table 10.
3. 3D SPECTRAL MAP DENOISING
In this experiment, we use two types of spectral moments: spherical harmonics and Zernike polynomials (see App. 4).
The minimum number of sample points required to accurately represent a ﬁnite energy function in a particular function space depends on the used sampling theorem.
According to
Driscoll and Healy’s theorem Driscoll & Healy (1994), 4 N 2 equiangular sampled points are needed to represent a function on S 2 using spherical moments at a maximum degree N .
Therefore, we compute the ﬁrst 16384 spherical moments of 3D objects where l ≤ 128 by sampling 256 ×equiangular points in θ and φ directions, where 0 ≤ θ ≤ π and 0 ≤ φ ≤ 2 π .
Afterwards, we arrange the spherical moments as a 128 × 128 feature map, and convolve with a 2 × 2 kernel with stride size 2 to downsample the feature map to 64 × 64 size.
The output is then fed to 64 -size architecture.
We add Gaussian noise and mask portions of the spectral map to corrupt it.
Afterwards, the model is trained to de-noise the input.
For Zernike polynomials, we compute the ﬁrst 100 moments for each 3D object where n ≤ 9 , and arrange the moments as a 10 × 10 feature map.
Then, the feature map is upsampled using transposed convolution by using a 5 × 5 kernel and with a stride size 3 .
The upsamapled feature map is fed to a 32 -size network and trained end-to-end to denoise.
We ﬁrst train the network on 55k objects in ShapeNet, and then apply the trained network on the Modelnet10 and Modelnet40 to extract the bottleneck features.
These features are then fed to a single fully connected layer for classiﬁcation.
S PECTRAL DOMAIN REPRESENTATION OF 3D OBJECTS
Spherical harmonics and Zernike polynomials are orthogonal and complete functions in S 2 and B 3 , respectively, hence, 3D point clouds can be represented by a set of coefﬁcients corresponding to a linear combination of these functions
Perraudin et al.
; Ramasinghe et al.
(2019a;c).
4. S PHERICAL HARMONICS
Spherical harmonics are complete and orthogonal functions deﬁned on the unit sphere ( S 2 ) as,
[Equation]
[Equation]
where θ ∈ [0 , 2 π ] is the azimuth angle, φ ∈ [0 , π ] is the polar angle, l ∈ Z + , m ∈ Z , and | m | < l .
Here, P ml ( · ) is the associated Legendre function deﬁned as,
[Equation]
[Equation]
Spherical harmonics demonstrate the following orthogonal property,
[Equation]
(27) where † denotes the complex conjugate and,
[Equation]
Since spherical harmonics are complete in S 2 , any function f : S 2 → R with ﬁnite energy can be rewritten as
[Equation]
where,
[Equation]
4. 3D Z ERNIKE POLYNOMIALS
3D Zernike polynomials are complete and orthogonal on B 3 and deﬁned as,
[Equation] [Equation]
where,
[Equation]
and q vnl is a scaler deﬁned as
[Equation]
Here Y l,m ( θ, φ ) is the spherical harmonics function, n ∈ Z + , l ∈ [0 , n ] , m ∈ [ − l, l ] and n − l is even.
3D Zernike polynomials also show orthogonal properties as,
[Equation]
Since Zernike polynomials are complete in B 3 , any function f : B 3 → R with ﬁnite energy can be rewritten as,
[Equation]
35) where Ω n,l,m ( f ) can be obtained using
[Equation]
I MAGE - TO -I MAGE TRANSLATION 5. S KETCH - TO - SHOES QUALITATIVE RESULTS
Additional qualitative results of the sketch-to-shoe translation task are shown in Fig.
5. M AP - TO - PHOTO QUALITATIVE RESULTS
Additional qualitative results of the map-to-photo translation task are shown in Fig.
C ONVERGENCE AT INFERENCE
A key aspect of our method is the optimization of the predictions at inference.
Fig.
40 and Fig.demonstrate this behaviour on the MNIST image completion and STL colorization tasks, respectively.