S TYLE - BASED E NCODER P RE - TRAINING FOR M ULTIMODAL I MAGE S YNTHESIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Image-to-image (I2I) translation aims to translate images from one domain to another.
To tackle the multi-modal version of I2I translation, where input and output domains have a one-to-many relation, an extra latent input is provided to the generator to specify a particular output.
Recent works propose involved training objectives to learn a latent embedding, jointly with the generator, that models the distribution of possible outputs.
Alternatively, we study a simple, yet powerful pre-training strategy for multi-modal I2I translation.
We ﬁrst pre-train an encoder, using a proxy task, to encode the style of an image, such as color and texture, into a low-dimensional latent style vector.
Then we train a generator to transform an input image along with a style-code to the output domain.
Our generator achieves stateof-the-art results on several benchmarks with a training objective that includes just a GAN loss and a reconstruction loss, which simpliﬁes and speeds up the training signiﬁcantly compared to competing approaches.
We further study the contribution of different loss terms to learning the task of multi-modal I2I translation, and ﬁnally we show that the learned style embedding is not dependent on the target domain and generalizes well to other domains.
I NTRODUCTION
Image-to-Image (I2I) translation is the task of transforming images from one domain to another (e.g., semantic maps → scenes, sketches → photo-realistic images, etc.).
Many problems in computer vision and graphics can be cast as I2I translation, such as photo-realistic image synthesis (Chen & Koltun (2017); Isola et al.
; Wang et al.
(2018a))
, super-resolution (Ledig et al.
, colorization (Zhang et al.
(2016; 2017a)), and inpainting (Pathak et al.
Therefore, I2I translation has recently received signiﬁcant attention in the literature.
One main challenge in I2I translation is the multi-modal nature for many such tasks – the relation between an input domain
A and an output domain B is often times one-to-many, where a single input image I Ai ∈
A can be mapped to different output images from domain B .
For example, a sketch of a shoe or a handbag can be mapped to corresponding objects with different colors or styles, or a semantic map of a scene can be mapped to many scenes with different appearance, lighting and/or weather conditions.
Since
I2I translation networks typically learn one-to-one mappings due to their deterministic nature, an extra input is required to specify an output mode to which an input image will be translated.
Simply injecting extra random noise as input proved to be ineffective as shown in (Isola et al.
; Zhu et al.
(2017b)), where the generator network just learns to ignore the extra noise and collapses to a single or few modes (which is one form of the mode collapse problem).
To overcome this problem, Zhu et al.
(2017b) proposed BicycleGAN , which learns to encode the distribution of different possible outputs into a latent vector z , and then learns a deterministic mapping G : ( A, z ) → B .
So, depending on the latent vector z , a single input I Ai ∈
A can be mapped to multiple outputs in B .
While BicycleGAN requires paired training data, several works (Lee et al.
; Huang et al.
(2018)) extended it to the unsupervised case, where images in domains A and B are not in correspondence (‘unpaired’).
One main component of unpaired I2I is a cross-cycle consistency constraint, where the network generates an intermediate output by swapping the styles of a pair of images, then swaps the style between the intermediate output again to reconstruct the original images.
This enforces that the latent vector z preserves the encoded style information when translated from an image i to another image j and back to image i again.
This constraint can also be applied to paired training data, where it encourages style/attribute transfer between images.
However, training BicycleGAN (Zhu et al. (2017b)) or trivial example,
BicycleGAN combines the objectives of both conditional Variational Auto-Encoders (cVAEs) (Sohn et al.
and a conditional version of Latent Regressor GANs (cLR-GANs) (Donahue et al.
; Dumoulin et al.
to train their network.
The training objective of (Huang et al.
Lee et al.
(2018)) is even more involved to handle the unsupervised setup.
In this work, we aim to simplify the training of general purpose multi-modal I2I translation networks, while also improving the diversity and expressiveness of different styles in the output domain.
Our approach is inspired by the work of Meshry et al.
(2019) which utilizes a staged training strategy to re-render scenes under different lighting, time of day, and weather conditions.
We propose a pretraining approach for style encoders, in multi-modal I2I translation networks, which makes the training simpler and faster by requiring fewer losses/constraints.
Our approach is also inspired by the standard training paradigm in visual recognition of ﬁrst pretraining on a proxy task, either large supervised datasets (e.g., ImageNet) (Krizhevsky et al.
; Sun et al.
; Mahajan et al.
(2018)) or unsupervised tasks (e.g., Doersch et al.
; Noroozi & Favaro (2016)), and then ﬁne-tuning (transfer learning) on the desired task.
Similarly, we propose to pretrain the encoder using a proxy task that encourages capturing style into a latent space.
Our goal is to highlight the importance of pretraining for I2I networks and demonstrate that a simple approach can be very effective for multi-modal image synthesis.
In particular, we make the following contributions:
• We explore style pretraining and its generalization for the task of multi-modal I2I translation, which simpliﬁes and speeds up the training compared to competing approaches.
• We provide a study of the importance of different losses and regularization terms for multi-modal I2I translation networks.
• We show that the pretrained latent embeddings is not dependent on the target domain and generalizes well to other domains (transfer learning).
• We achieve state-of-the art results on several benchmarks in terms of style capture and transfer, and diversity of results.
R ELATED WORK
Deep generative models There has been incredible progress in the ﬁeld of image synthesis using deep neural networks.
In its unconditional setting, a decoder network learns to map random values drawn from a prior distribution (typically Gaussian) to output images.
Variational Auto-Encoders (VAEs) (Kingma & Welling (2014)) assume a bijection mapping between output images and some latent distribution and learn to map the latent distribution to a unit Gaussian using the reparameterization trick.
Alternatively, Generative Adversarial Networks (GANs) (Goodfellow et al.
(2014)) directly map random values sampled from a unit Gaussian to images, while using a discriminator network to enforce that the distribution of generated images resembles that of real images.
Recent works proposed improvements to stabilize the training (Gulrajani et al.
; Karnewar & Iyengar (2019); Mao et al.
; Radford et al.
and improve the quality and diversity of the output (Karras et al.
Other works combine both VAEs and GANs into a hybrid VAE-GAN model (Larsen et al.
; Rosca et al.
Conditional image synthesis Instead of generating images from input noise, the generator can be augmented with side information in the form of extra conditional inputs.
For example, Sohn et al.
(2015) extended VAEs to their conditional setup (cVAEs).
Also, GANs can be conditioned on different information, like class labels (Mirza & Osindero (2014); Odena et al.
(2017); Van den Oord et al.
language description (Mansimov et al.
; Reed et al.
(2016)), or an image from another domain (Chen & Koltun (2017); Isola et al.
The latter is called Image-to-Image translation.
Image-to-Image (I2I) translation I2I translation is the task of transforming an image from one domain, such as a sketch, into a another domain, such as photo-realistic images.
While there are regression-based approaches to this problem (Chen & Koltun (2017); Hoshen & Wolf (2018)), signiﬁcant successes in this ﬁeld are based on GANs and the inﬂuential work of pix2pix (Isola et al.
Following the success of pix2pix (Isola et al.
(2017)), I2I translation has since been utilized in a large number of tasks, like inpainting (Pathak et al.
, colorization (Zhang et al.
(2016; 2017a)), super-resolution (Ledig et al.
, rendering (Martin-Brualla et al.
; Meshry et al.
; Thies et al.
and many more (Dong et al.
; Wang & Gupta (2016); Zhang et al.
(2017b)).
There has also been works to extend this task to the unsupervised setting (Hoshen & Wolf
[Figure]
; Kim et al.
; Liu et al.
; Ma et al.
; Royer et al.
; Zhu et al.
(2017a)), to multiple domains (Choi et al.
(2018)), and to videos (Chan et al.
; Wang et al.
(2018b)).
Multi-modal I2I translation Image translation networks are typically deterministic function approximators that learn a one-to-one mapping between inputs and outputs.
To extend I2I translation to the case of diverse multi-modal outputs, Zhu et al.
(2017b) proposed the BicycleGAN framework that learns a latent distribution that encodes the variability in the output domain and conditions the generator on this extra latent vector for multi-modal image synthesis.
Wang et al.
(2018a;b) learn instance-wise latent features for different objects in a target image, which are clustered after training to ﬁnd f ﬁxed modes for different semantic classes.
At test time, they sample one of the feature clusters for each object to achieve multi-modal synthesis.
Other works extended the multi-modal
I2I framework to the unpaired settings where images from the input and output domains are not in correspondence (Almahairi et al.
; Huang et al.
; Lee et al.
by augmenting
BicycleGAN with different forms of a cross-cycle consistency constraint between two unpaired images.
In our work, we focus on the supervised setting of multi-modal I2I translation.
We propose a simple, yet effective, pretraining strategy to learn a latent distribution that encodes variability in the output domain.
The learned distribution can be easily adapted to new unseen datasets with simple ﬁne-tuning, instead of training from random initialization.
A PPROACH
Current multi-modal image translation networks require an extra input z that allows for modelling the one-to-many relation between an input domain A and an output domain B as a one-to-one relation from a pair of inputs ( A, z ) → B .
In previous approaches, there has been a trade-off between simplicity and effectiveness for providing the input z .
On one hand, providing random noise as the extra input z maintains a simple training objective (same as in pix2pix (Isola et al.
However, Isola et al.
; Zhu et al.
(2017b) showed that the generator has little incentive to utilize the input vector
z since it only encodes random information, and therefore the generator ends up ignoring z and collapsing to one or few modes.
On the other hand, BicycleGAN (Zhu et al. (2017b)) combines the objectives of both conditional Variational Auto-Encoder GANs (cVAE-GAN) and conditional Latent Regressor GANs (cLR-GAN) to learn a latent embedding z simultaneously with the generator G .
Their training enforces two cycle consistencies: B → z → ˆ B and z → ˜ B → ˆ z .
This proved to be very effective, but the training objective is more involved, which makes the training slower.
Also, since the latent embedding is being trained simultaneously with the the generator, hyper-parameter tuning becomes more critical and sensitive.
We aim to combine the best of both worlds: an effective training of a latent embedding that models the distribution of possible outputs, while retaining a simple training objective.
This would allow for faster and more efﬁcient training, as well as less sensitivity to hyper-parameters.
We observe that the variability in many target domains can be represented by the style diversity of images in the target domain B , where the style is deﬁned in terms of the gram matrices used in the Neural
Style Transfer literature (Gatys et al.
Then, we learn an embedding by separately training an encoder network E on an auxiliary task to optimize for z = E ( I B ) capturing the style of an image
I B .
Finally, since we now have learned a deterministic mapping between z and the style of the target
[Figure]
output image I B , training the generator G becomes simpler as G is just required to discover the correlation between output images and their corresponding style embedding z .
To incorporate this into BicycleGAN (Zhu et al. (2017b)), we replace the simultaneous training of the encoder E and the generator G with a staged training as follows:
Stage 1: Pretrain E on a proxy task that optimizes an embedding of images in the output domain B into a low-dimensional style latent space, such that images with similar styles lie closely in that space (i.e. clustered).
Stage 2: Train the generator network G while ﬁxing the encoder E , so that G learns to associate the style of output images to their deterministic style embedding z = E
( I B ) .
Stage 3: Fine-tune both the E and G networks together, allowing for the style embedding to be further adapted to best suit the image synthesis task for the target domain.
The intuition why such staged training would be effective for multi-modal I2I translation is that the encoder is pretrained to model different modes of the output distribution as clusters of images with similar styles (refer to the supp.
material, ﬁgures 6,7, for a visualization of pretrained latent embeddings).
During stage 2, the latent space is kept ﬁxed, and the input latent to the generator can be used to clearly distinguish the style cluster to which the output belongs, which makes the multi-modal synthesis task easier for the generator.
Finally, stage 3 ﬁnetunes the learned embedding to better serve the synthesis task at hand.
Next, we explain how to pre-train the style encoder network
E in Section 3.1, and how to train the generator G using the pre-learned embeddings (Section 3.2).
Finally, we demonstrate the generalization of pre-training the style encoder E in Section 3.3.
3. P RE - TRAINING THE STYLE ENCODER E
The goal of pre-training the encoder network E is to learn a deterministic mapping from the style of a target image I Bi ∈ B to a latent style code z i
=
E
( I Bi ) .
Ideally, images with similar styles should be close in the style embedding space, while images with different styles should be far from each other.
To supervise training such embedding, we utilize the style loss (Gatys et al.
as a distance metric to measure the style similarity between any two given images.
The style encoder network
E is then trained using a triplet loss (Schroff et al.
(2015)), where the input is a triplet of images
( I a , I p , I n ) , where ( I a , I p ) have similar style, while ( I a , I n ) have different style, as measured by the style loss metric.
The training objective for E is given by:
[Equation]
where α is a separation margin, λ is a relative weighting parameter between the main triplet objective and an optional regularization term L reg ( · ) which is an L 2 regularization to encourage learning a compact latent space.
Triplet selection.
To generate triplets for pre-training the encoder E , we compute the set of k c closest and k f furthest neighbors for each anchor image I a as measured by the style loss.
Then, for each anchor image I a , we randomly sample a positive image
I p and a negative image I n from the set of closest and furthest neighbors respectively.
We found that, for large datasets, it is sufﬁcient to generate triplets for a subset of the training images.
One challenge is the set of images with an outlier style.
Such images will be furthest neighbors to most images, and can mislead the training by just projecting outlier images to separate clusters.
To deal with this, we sample the negative style image
I n from a larger set of furthest neighbors; while the positive image I p is sampled from a small set of closest neighbors so that it would have reasonable style similarity to the anchor image.
3. G ENERATOR TRAINING
After pre-training the style encoder E (stage 1), we have established a mapping from images in the output domain, I B ∈ B , to their style-embedding z
= E ( I B ) .
Feeding the style embedding as input to the generator during training, the generator has good incentive to associate the style of output images to their corresponding style embedding instead of learning to hallucinate the style.
It’s important to retain the deterministic correspondence between images and their style codes to facilitate the job of the generator to discover this correlation.
This is why, during stage 2, we keep the weights of the style encoder E ﬁxed.
The forward pass reconstructs a training image
I Bi as ˆ I Bi =
G ( I Ai , z i ) , where z i =
E
( I Bi ) .
The training objective is similar to that of pix2pix (Isola et al.
[Equation]
where we use the Least Square GAN loss (LSGAN) (Mao et al.
for the L cGAN term, and a VGG-based perceptual loss (Johnson et al.
for the reconstruction term L rec .
Once the generator has learned to associate the output style with the input style embedding, stage 3 ﬁne-tunes both the generator G and the style encoder E together using the same objective (2).
Style sampling.
To perform multimodal synthesis on a given input at test time, we can capture the latent vector z from any existing image and transfer the style to the generated image.
However, if we wish to sample styles directly from the latent distribution, we can optionally enforce a prior on the latent distribution.
For example, we can add a KL divergence term on the latent vectors to enforce a unit Gaussian prior.
In our experiments, we found it more effective to add an L 2 regularization on the the latent vectors to enforce zero-mean embeddings and limit the variance of the latent space.
We then compute an empirical standard deviation for sampling.
Another alternative to enable sampling is to train a mapper network M to map the unit Gaussian to the latent distribution.
This can be done as a post-processing step after the style encoder has been trained and ﬁnetuned.
Speciﬁcally, we follow the nearest-neighbor based Implicit Maximum Likelihood Estimation (IMLE) training (Li & Malik (2018); Hoshen et al.
to train the mapper network M .
The training objective is given by:
[Equation]
where { r j } is a set of random samples from the unit Gaussian prior, and for each latent code z i , we select e i that maps to the nearest neighbor M ( e i ) to z i .
[Figure]
s 2 s hoe s edge s
2handbag s m ap s s pa c e need l e n i gh t 2da y l abe l s 2 f a c ad s
3. G ENERALIZING THE PRE - TRAINING STAGE
The goal of the style pretraining is to learn an embedding that mimics the style loss, where images with similar style lie closely in that space.
Since the deﬁnition of image style in neural style transfer literature is general and is not dependent on a speciﬁc image domain, encoding an image I to its style embedding can also be seen as a general task that is independent of the output domain B .
This allows for performing the pretraining stage only once using auxiliary training data.
The ﬁnetuning stage eventually tweaks the embedding to better suit the speciﬁc target domain B .
We show experimentally in Section 4 that pretraining the style encoder on datasets other than the target domain B doesn’t degrade the performance.
It can even improve the performance if the target dataset is small, in which case pretraining on an auxiliary dataset helps with the generalization of the overall model.
E XPERIMENTAL EVALUATION
Datasets.
We evaluate our approach on ﬁve standard I2I translation benchmarks used in Isola et al.
(2017); Zhu et al.
(2017b)
; Architectural labels → photo, aerial → map, edges → shoes/handbags and night → day.
In addition, we use the Space Needle timelapse dataset (Martin-Brualla, 2007), which consists of 2068 paired images with a 8280 × 1080 resolution, where the input domain includes images with temporally smoothed appearance, and the output domain contains real images spanning different lighting and weather conditions.
Baselines.
While we report numbers for retrained models using the code released with BicycleGAN
(BicycleGAN v0)
for completeness, we mainly compare to two stronger baselines:
• BicycleGAN v1 : we implement BicycleGAN using the same network architecture as used in our approach to have a fair comparison.
• BicycleGAN v2 : we augment BicycleGAN with the cross-cycle consistency constraint introduced in (Huang et al., 2018; Lee et al., 2018) as follows: the input is a pair of training examples ( I A 1 , I B 1 ) , ( I A 2 , I B 2 ) for which we obtain their respective style embeddings z 1 =
E
( I B 1 ) , z
2 = E ( I B 2 ) .
We then apply a 2-step cyclic reconstruction of I B 1 , I B 2 ; in the ﬁrst step, we generate both images with a swapped style u =
G ( I A 1 , z 2 ) , v = G
( I A 2 , z 1 ) .
In the the second step, we re-capture the latent style vectors ˆ z 2 = E ( u ) , ˆ z 1 = E ( v ) and generate the original images I B 1 , I B 2 by swapping the style again: ˆ I B 1 =
G
( I A 1 , ˆ z 1 ) , ˆ I B 2 =
G ( I A 2 , ˆ z 2 ) .
We add a cyclic reconstruction term for ˆ I B 1 , ˆ I B 2 .
4. E VALUATION
Image reconstruction.
We report the reconstruction quality of validation set images, using both
PSNR and AlexNet -based LPIPS (Zhang et al., 2018)
metrics, in Table 1.
Note that our results without ﬁne-tuning (stage 2) are on-par-with the baselines, which veriﬁes the validity of our approach and
[Table]
with different datasets.
Stages 2, 3 show results before/after ﬁnetuning E respectively.
[Table] [Table]
that style-based encoder pre-training successfully learns to distinguish different modes in the output domain, which proves effective for training multi-modal I2I networks.
Fine-tuning (stage 3) further improves our results compared to the baselines.
Figure 2 shows qualitatively how our approach reconstructs the target style more faithfully.
Style transfer and sampling.
Figure 3 shows style transfer to validation set images from different datasets.
We can also sample random styles directly from the the latent distribution as described in
Section 3.2.
Figure 5 shows results for both adhoc sampling from the assumed N ( µ, σ ) empirical distribution, as well as formally sampling from a unit Gaussian using the mapper network M .
While both results look good, we note that the assumption for adhoc sampling is not explicitly enforced, and thus could sample bad style codes outside the distribution (see Appendix A.5 for examples).
Style interpolation.
Figure 4 shows style interpolation by linearly interpolating between two latent vectors.
Note the smooth change in lighting and cloud patterns when going from cloudy to sunny (second row).
Pre-training generalization.
Since the notion of style, as deﬁned in the Neural Style Transfer literature, is universal and not speciﬁc to a certain domain, we hypothesized that style-based encoder pretraining would learn a generic style embedding that can generalize across multiple domains and be effective for multi-modal image I2I translation.
Here we experimentally verify our hypothesis in Table 2.
For a target dataset, we train the generator G three times, each with different pre-training of the style encoder E : (1) same dataset pre-training: pre-train E using the output domain B of the target dataset.
(2) similar-domain pre-training: pre-train on a different dataset, but whose output domain bears resemblance to the output domain of the target dataset (e.g., edges2shoes and edges2handbags, or day images from night2day and the Space Needle timelapse dataset).
(3) different-domain pretraining: pre-train on a different dataset whose output domain has different styles from that of the target dataset (e.g., edges2handbags and the Space Needle timelapse datasets, or night2day and edges2handbags datasets).
Table 2 shows that without ﬁne-tuning (stage 2), the edges2handbags dataset shows a slight performance degradation when going from pre-training on the same dataset, to pre-training on a similar-domain dataset, and ﬁnally pre-training on a different-domain dataset.
On the other hand, the night2day dataset has only ∼ 100 unique scenes for training.
So, pre-training on another dataset such as Space Needle generalizes better to new scenes in the validation set, since it helps avoid overﬁtting the small number of unique scenes in the training set.
After ﬁne-tuning, [Table] performance differences further reduce to be insigniﬁcant.
We also investigate the generalization of encoder pre-training using non-style distance metrics in Appendix A.4.
Ablative study.
We investigate the role of different loss terms as we transition from the loss setup of baselines to that of our training approach.
We ﬁrst remove the variational part in both BicycleGAN v and BicycleGAN v2 baselines resulting in Bicycle v1.2, v2.2.
We further remove the Gaussian prior and replace the KL loss with an L2 regularization in Bicycle v1.3, v2.3.
To maintain random latent vector sampling during training without a prior, we sample a random training image, and use its style code.
We deﬁne different versions of our approach (v1, v2, v3, v4) based on different losse setup during training as follows: we start with ‘Ours v1’, which has the same setup as Bicycle v2.3, except that it uses pre-trained embeddings as described in Section 3.1.
We then remove cyclic reconstruction, random z sampling, and L2 regularization terms resulting in ‘Ours v2’, ‘v3’, and ‘v4’ respectively.
We run each setup on the edges2handbags dataset.
In order to draw more reliable conclusions, we repeat each experiment 3 times and report the mean and standard deviation in Table 4.
We notice that removing the variational part in VAEs is enough to achieve good results.
While VAEs in general are robust to noise in the input latent, we observe that this comes at the expense of the expressiveness of the latent space (e.g., less faithful style capture and transfer), especially for low dimensional latents.
We also observe that our approach generally performs better with less constraints (loss terms).
For example, “Ours v1, v2” have lower results than their “Bicycle v1.3, v2.3” counterparts.
This shows that the main beneﬁt of pre-trained embeddings is when the network is less constrained.
Diversity and user study.
We evaluate diversity by computing the average LPIPS distance over 1600 output images.
We measure diversity on two setups: we sample 100 validation images, and (1) apply style transfer from 16 randomly sampled images, or (2) we sample 16 random codes using the mapper network M to obtain 1600 outputs.
We also measure the realism and faithfulness of style transfer through a user study, where 30 participants are shown an input shoe sketch, an input style image and two style transfer outputs.
They are given unlimited time to choose which output looks more realistic, and if both are realistic, then which transfers the style more faithfully.
We ﬁx ‘Ours v4’ approach as anchor and compare other methods to it.
Table 3 shows that the baselines achieve lower diversity and user preference compared to our approach, specially in the style transfer setup.
Different variations of our method, except for ‘Ours v2’ yield similar diversity and user preference scores.
We observe that ‘Ours v2’ shows artifacts in some outputs, leading to higher diversity but lower user preference.
Our diversity results for the style sampling setup have some variation and are sensitive the mapper network training, but are still either on-par or better than the baselines.
Conclusion.
We investigated the effectiveness of embedding pre-training for the task of multimodal I2I translation.
The pre-training can be done once on auxiliary data, and generalizes well to other domains.
This allows for a faster training of I2I translation networks with fewer losses and achieves more faithful style capture and transfer.
Furthermore, we studied the contribution of different loss terms, where we discovered that noise added by a variational auto-encoder can limit the expressiveness of low-dimensional latent spaces.
Finally, we achieved state-of-the-art results on several benchmarks.
A A PPENDIX A. I MPLEMENTATION DETAILS
The generator network G has a symmetric encoder-decoder architecture based on Wang et al.
(2018a), with extra skip connections by concatenating feature maps of the encoder and decoder.
We use a multiscale-patchGAN discriminator (Wang et al., 2018a) with 3 scales and employ a LSGAN (Mao et al., 2017) loss.
The mapper network M is a multi-layer perceptron (MLP) with three 128dimensional hidden layers and a tanh activation function.
For the reconstruction loss, we use the perceptual loss (Johnson et al., 2016) evaluated at conv i, 2 for i ∈
[1 , 5] of VGG (Simonyan &
Zisserman, 2014) with linear weights of w i = 1 / 2 6 − i for i ∈ [1 , 5] .
The architecture of the style encoder E is adopted from Lee et al.
and we use a latent style vector z ∈ R 8 .
Our optimizers setup is similar to that in Zhu et al.
(2017b).
We use three Adam optimizers: one for the generator G and encoder E , another for the discriminator D , and another optimizer for the generator G alone with β 1 = 0 , β 2 = 0 .
99 for the three optimizers, and learning rates of 0 .
001 and 0 .
respectively.
We use a separate Adam optimizer for the mapper network M with β 1 = 0 .
5 , β 2 = 0 .
99 , and a learning rate of 0 .
01 with a decay rate of 0.7 applied every 50 steps.
Relative weights for the loss terms are λ cGAN = 1 , λ rec = 0 .
02 and λ L 2 = 0 .
01 for the GAN loss, reconstruction loss, and Llatent vector regularization respectively.
When sampling triplets for any anchor image I c , we use k c = 5 , k f = 13 for the size of the set of close and far neighbors respectively.
A. M ORE QUANTITATIVE COMPARISON
We report the Inception Score (IS) computed over the validation set of various datasets in Table 5.
Surprisingly, results after ﬁnetuning (“ours - stage 3”) are slightly worse than those before ﬁnetuning (“ours - stage 2”), but both are still better than the baselines except for the maps dataset.
We also note the Inception Score is not very suited to image-to-image translation tasks, since it prefers output diversity with respect to ImageNet classes, not within-class diversity as in our case.
[Table]
A. L ATENT SPACE VISUALIZATION
Figure 6a visualizes the latent space learned by the style encoder E after pretraining and before ﬁnetuning (a), after ﬁnetuning (b), and the latent space learned by BicycleGAN (Zhu et al., 2017b)
(c).
The embedding learned through pretraining and before ﬁnetuning shows meaningful clusters.
Finetuning further brings the embedding closer to that of BicycleGAN.
A. E NCODER PRE - TRAINING WITH NON - STYLE METRICS
Pre-training the encoder using a style-based triplet loss showed to be successful for multi-modal image translation tasks where the variability in the target domain is mainly color-based.
This is shown in the results obtained on several benchmarks, even before the ﬁnetuning stage (“ours - stage 2” in Table 1).
We note though that the usage of style-loss as a distance metric for triplet sampling is just one choice and can be replaced with other distance metrics depending on the target application.
Triplet sampling with style distance results in learning an embedding space where images with similar colors/styles lie closely in that space as shown in Section A.3.
If, for example, we sample triplets instead based on the distance between VGG-Face (Parkhi et al., 2015) embeddings, the encoder will learn a latent space which is clustered by identity.
In this section, we aim to validate that the proposed pre-training strategy can be extended to multi-modal image-to-image translation tasks with non-style variability.
We inspect the task of manipulating facial expressions, where the input is a neutral face, and the output can have other emotions or facial expressions.
For this task, similar emotions should be embedded closely in the latent space.
We therefore use an off-the-shelf facial expression recognition system to compute the emotion similarity/distance between any pair of images.
Speciﬁcally, we compute the emotion distance as the euclidean distance between the 512-dimensional feature map of the last layer of a pretrained classiﬁcation network (e.g., (Jie, 2018)).
We visualize the learned latent space in Figure 7, which shows clusters with similar emotions or facial expressions.
We also show example translation results on a holdout set of the front-view images of the KDEF dataset (KDEF, 2017) in Figure 8.
We note that the generator successfully learns to manipulate facial expressions based solely on the pre-trained embeddings (without the ﬁnetuning stage).
On the other hand, the BicycleGAN-based baselines collapsed to a single mode (over 3 different runs).
This shows that our staged-training approach is stable and not sensitive to hyper-parameters, unlike the
BicycleGAN baselines which will require careful hyper-parameter tuning to work properly on this task.
We also point out that the poor output quality is mainly due to using a pixel-wise reconstruction loss for the generator training, while the input-output pairs in this dataset are not aligned.
We didn’t investigate improving the generator training as this is orthogonal to verifying the generalization of encoder pre-training.
A. S TYLE SAMPLING COMPARISON
Figure 9 compares style sampling using the mapper network M vs adhoc sampling from the assumed
N ( µ, σ ) of an L 2 -regularized latent space, where µ, σ are empirically computed from the training set.
Note that adhoc sampling can sometimes sample bad style codes outside the distribution (e.g. third image in ﬁrst row, and ﬁrst image in third row in the right side of Figure 9), since the assumption that a L 2 -regularized space would yield normally distributed latents with zero mean and low standard deviation is not explicitly enforced.
A. T RAINING TIME
Simplifying the training objective allows for faster training time, as well as a larger batch size due to lower memory usage.
Table 6 shows the processing time per 1000 training images for the baselines as well as different variations of our approach as deﬁned in Table 4.
[Table]
A. C ONVERGENCE ANALYSIS
Figure 10 compares the convergence of our staged training compared to the BicycleGAN baselines.
The dotted line in the graph marks the transition between stages 2 and 3 of our training (i.e, switching from a ﬁxed pre-trained encoder E to ﬁnetuning both G and E together).
We measure the reconstruction error (LPIPS) of the validation set of the edges2handbags dataset as the training progresses.
Results show that with a ﬁxed pre-trained encoder, staged training starts with higher error than the baselines, but quickly drops to show similar performance as the baselines, and even beats the baselines before switching to stage 3 (marked by a dotted line).
When starting to ﬁnetune the encoder
E , we get a spike in the reconstruction error but then the performance of our staged training steadily widens the performance gap with the baselines.
This shows the importance of the ﬁnetuning stage to tweak the pre-trained embeddings to better serve the image synthesis task.
[Figure]
[Figure] [Figure]