L ATENT D OMAIN T RANSFER : C ROSSING MODALITIES WITH B RIDGING A UTOENCODERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Domain transfer is an exciting and challenging branch of machine learning because models must learn to smoothly transfer between domains, preserving local variations and capturing many aspects of variation without labels.
However, most successful applications to date require the two domains to be closely related (e.g., image-to-image, video-video), utilizing similar or shared networks to transform domain-speciﬁc properties like texture, coloring, and line shapes.
Here, we demonstrate that it is possible to transfer across modalities (e.g., image-to-audio) by ﬁrst abstracting the data with latent generative models and then learning transformations between latent spaces.
We ﬁnd that a simple variational autoencoder is able to learn a shared latent space to bridge between two generative models in an unsupervised fashion, and even between different types of models (e.g., variational autoencoder and a generative adversarial network).
We can further impose desired semantic alignment of attributes with a linear classiﬁer in the shared latent space.
Through qualitative and quantitative evaluations, we demonstrate that the proposed model preserves both locality and semantic alignment through the transfer process.
Finally, the hierarchical structure decouples the cost of training the base generative models and semantic alignments, enabling computationally efﬁcient and data efﬁcient retraining of personalized mapping functions.
I NTRODUCTION
Domain transfer has long captured the imagination of inventors and artists alike.
The early precursor of the phonograph, the phonautograph, was actually inspired by the idea of ”words which write themselves”, where the shape of audio waveforms would transform into the shape of writing, capturing the content and character of the speaker’s voice in shape and stroke of the written characters (Feaster, 2009).
While perhaps fanciful at the time, modern deep learning techniques have shown similar complex transformations are indeed possible.
Deep learning enables domain transfer by learning a smooth mapping between two domains such that the variations in one domain are reﬂected in the other.
This has been demonstrated to great effect within a data modality, for example transferring between two different styles of image (Isola et al., 2016; Zhu et al., 2017; Li et al., 2018; Li, 2018), video (Wang et al., 2018) and music (Mor et al., 2018).
The works have been the basis of interesting creative tools, as small intuitive changes in the source domain are reﬂected by small intuitive changes in the target domain.
Furthermore, the strong conditioning signal of the source domain makes learning transformations easier than learning a full generative model in each domain.
Despite these successes, this line of work in domain transfer has several limitations.
The ﬁrst limitation is that it requires that two domains should be closely related (e.g. image-to-image or videoto-video).
This allows the model to focus on transferring local properties like texture and coloring instead of high-level semantics. For example, directly applying these image-to-image transfer such as CycleGAN (Zhu et al., 2017) or its variants to images from distant domains leads to distorted and unrealistic results (Li et al., 2018). This agrees with the ﬁndings of Chu et al.
(2017) who show that
CycleGAN transformations are more akin to adversarial examples than style transfer, as the model
(b) Domain Transfer q(z 1 |x 1 ) q(z'|z,D=1) xxg(z|z',D=2)
g(z 2 ) z' zz
Our Model
Pre-trained Model
Data Space
Latent Space
Shared Latent Space Loss LEGEND q(z 1 |x 1 ) q(z'|z,D=1)
xg(z|z',D=1) z' zz(a)
Training
L q(z 2 |x 2 ) q(z'|z,D=2) xg(z|z',D=2)
z' zz 2 L
SWD
KL + Classifier
KL + Classifier to ﬁrst we to a shared latent vector z (cid:48) using our conditional encoder, z (cid:48) ∼ q ( z (cid:48) | z 1 , D = 1) , where D donates the operating domain.
We then decode to the latent space of the target domain z 2 =
g ( z | z (cid:48) , D = 2) using our conditional decoder, which ﬁnally is used to generate the transferred audio x 2 = g ( x 2 | z 2 ) .
learns to hide information about the source domain in near-imperceptible high-frequency variations of the target domain. The second limitation is data efﬁciency.
Most conditional GAN techniques, such as pix2pix (Isola et al., 2016) and vid2vid (Wang et al., 2018), require very dense supervision from large volumes of paired data.
This is usually accomplished by extracting features, such as edges or a segmentation map, and then training the conditional GAN to learn the inverse mapping back to pixels.
For many more interesting transformations, no such easy alignment procedure exists, and paired data is scarce.
For multi-modal domain transfer, we seek to train a model capable of transferring instances from a source domain ( x 1 ) to a target domain ( x 2 ), such that local variations in source domain are transferred to local variations in the target domain.
We refer to this property as locality .
Thus, local interpolation in the source domain would ideally be similar to local interpolation in target domain when transferred.
There are many possible ways that two domains could align such that they maintain locality, with many different alignments of semantic attributes.
For instance, for a limited dataset, there is no a priori reason that images of the digit “0” and spoken utterances of the digit “0” would align with each other.
Or more abstractly, there may be no agreed common semantics for images of landscapes and passages of music, and it is at the liberty of the user to deﬁne such connections based on their own intent.
Our goal in modeling is to respect the user’s intent and make sure that the correct semantics (e.g., labels) are shared between the two domains after transfer.
We refer to this property as semantic alignment .
A user can thus sort a set of data points from in each domain into common bins, which we can use to constrain the cross-domain alignment.
We can quantitatively measure the degree of semantic alignment by using a classiﬁer to label transformed data and measuring the percentage of data points that fall into the same bin for the source and target domain.
Our goal can thus be stated as learning transformations that preserve locality and semantic alignment, while requiring as few labels from a user as possible.
To achieve this goal and tackle prior limitations, we propose to abstract the domain domains with independent latent variable models, and then learn to transfer between the latent spaces of those models.
Our main contributions include:
• We propose a shared ”bridging” VAE to transfer between latent generative models. Locality and semantic alignment of transformations are encouraged by applying a sliced-wasserstein distance, and a classiﬁcation loss respectively to the shared latent space. • We demonstrate with qualitative and quantitative results that our proposed method enables transfer both within a modality (image-to-image) and between modalities (image-to-audio). • Since we training a smaller secondary model in latent space, we ﬁnd improvements in training efﬁciency, measured by both in terms of the amount of required labeled data and well training time.
Figure 1 diagrams our hierarchical approach.
We ﬁrst independently pre-train separate generative models, either a VAE or GAN, for both the source and target domain.
For VAEs, data is encoded from the data domain to the latent space through a learned encoding function z ∼ q ( z | x ) , and decoded back to the data space with a decoder function ˆ x ∼ g ( x | z ) .
For GANs, we choose latent samples from a spherical Gaussian prior z ∼ p ( z ) and then use rejection sampling to only select latent samples whose associated data x = g ( z ) is classiﬁed with high conﬁdence by an auxiliary classiﬁer.
We then add the bridging conditional VAE with shared weights, tasked with modeling both latent spaces z 1 , z 2 .
The VAE has a single shared latent space z (cid:48) , that corresponds to both domains.
Sharing the weights encourages the model to seek common structures between the latent domains, but we also ﬁnd it helpful to condition both the encoder q shared ( z (cid:48) | z, D ) and decoder g shared ( z | z (cid:48) , D ) , with an additional one-hot domain label, D , to allow the model some ﬂexibility to adapt to variations particular to each domain.
While the low-level VAEs have a spherical Gaussian prior, we penalize the KL-Divergence to be less than 1, allowing the models to achieve better reconstructions and retain some structure of the original dataset for the bridging VAE to model.
Full architectural and training details can be found in the Appendix. The conditional bridging VAE objective consists of three types of loss loss terms:
Evidence Lower Bound (ELBO) .
Standard value that is maximized for training a VAE. For each domain d ∈ { 1 , 2 } ,
[Equation]
where the likelihood π ( z ; g ) is a spherical Gaussian N ( z ; g, σ 2 I ) , and σ and β KL are hyperparmeters set to 1 and 0.1 respectively to encourage reconstruction accuracy.
Sliced Wasserstein Distance (SWD) (Bonneel et al., 2015). The distribution distance between mini-batches of samples from each domain in the shared latent space ( z (cid:48) 1 , z
[Equation]
[Equation]
[Equation]
where Ω is a set of random unit vectors, proj( A, a ) is the projection of A on vector a
and W 22 ( A, B ) is the quadratic Wasserstein distance.
Classiﬁcation Loss (Cls) . For each domain d ∈ { 1 , 2 } , we enforce semantic alignment with attribute labels y and a classiﬁcation loss in the shared latent space:
[Equation]
where H is the cross entropy loss, f ( z (cid:48) ) is a one-layer linear classiﬁer. [Figure]
Including terms for both domains, the total training loss is then
[Equation]
Where β SWD and β Cls are scalar loss weights.
The transfer procedure is illustrated Figure 2 using synthetic data.
For reconstructions, data x 1 is passed through two encoders, z 1 ∼ q ( z 1 | x 1 ) , z (cid:48) ∼ q shared ( z (cid:48)
| z 1 , D = 1) , are reconstructed through two decoders, ˆ z 1 ∼ g shared ( ˆ z 1 | z (cid:48) , D = 1) , ˆ x 1 ∼ g ( ˆ x 1 | ˆ z 1 ) .
For transformations, the encoding is the same, but decoding uses decoders (and conditioning) from the second domain, ˆ z 2 ∼ g shared ( ˆ z 2 | z (cid:48) , D = 2) , ˆ x 2 ∼ g ( ˆ x 2 | ˆ z 2 ) .
Further analysis of this example and intuition behind loss terms is summarized in Figure 2 and detailed in Appendix A.
E XPERIMENTS 3. D ATASETS
While the end goal of our method is to enable creative mapping between datasets with arbitrary alignments, for quantitative studies we restrict ourselves to three domains where there exist a somewhat natural alignment to compare against:
MNIST (LeCun, 1998), which contains images of hand-written digits of 10 classes from “0” to “9”.
Fashion MNIST (Xiao et al., 2017), which contains fashion related objects such as shoes, tshirts, categorized into 10 classes.
The structure of data and the size of images are identical to MNIST.
SC09 , a subset of Speech Commands Dataset 1 , which contains the record of audio of humans saying digits from “0” to “’9”.
For MNIST and Fashion MNIST, we prepare VAE with MLP encoder and decoder following setting up in Engel et al.
More speciﬁcally, we use stacks of fully-connected linear layers activated by ReLU, together with a “Gated Mixing Layer”.
The full network architecture of the bridging
VAE is detailed in Appendix B. For SC09 we use the publicly available WaveGAN (Donahue et al., 2018) 2 .
We would like to emphasize that we only use class level supervision for enforcing semantic alignment with the latent classiﬁer.
We examine three scenarios of domain transfer:
MNIST ↔ MNIST.
We ﬁrst train two lower-level VAEs from different initial conditions. The bridging autoencoder is then tasked with transferring between latent spaces while maintaining the digit class from source to target.
MNIST ↔ Fashion MNIST.
In this scenario, we specify a global one-to-one mapping between 10 digit classes and 10 fashion object classes (See Table 5 in Appendix for details). The bridging autoencoder is tasked with preserving this mapping as it transfers between images of digits and clothing.
MNIST ↔ SC09.
For the speech dataset, we ﬁrst train a GAN to generate audio waveforms (Donahue et al., 2018) of spoken digits.
We chose to use a WaveGAN because we wanted a global latent variable for the full waveform (as opposed to a distributed latent code as in Engel et al. It also gives us an opportunity to explore transferring between different classes of models.
The bridging autoencoder is then tasked with transferring between a VAE of written digits and a GAN of spoken digits.
3. R ECONSTRUCTION AND D OMAIN T RANSFER
For reconstructions and domain transfer, we present both qualitative and quantitative results.
Quantitative measurements of semantic alignment are performed with pre-trained classiﬁers in each data domain. Given that the datasets have pre-aligned classes, when evaluating transferring from data x dto x d 2 , the reported accuracy is the portion of instances that x d 1 and x d 2 have the same predicted class.
Qualitative reconstruction results are shown in Figure 3 and the quantitative reconstruction accuracies are given in Table 1. For domain transfer, qualitative results are shown in Figure 4 and the quantitative transfer accuracies are given in Table 2.
For both reconstruction and domain transfer, we see very good semantic alignment in the MNIST → MNIST and MNIST → Fashion MNIST, and reasonably good alignment in MNIST → SC09.
The lower performance on SC09 is likely due to the comparatively poor performance of the base generative model, and would improve with a better audio generation model.
For reference, the WaveGAN has to model a much larger signal than the MNIST VAEs (16000 dimensions vs.dimensions).
[Table]
3. I NTERPOLATION
Interpolation can act as a good proxy for locality and local smoothness of the latent transformations, as by deﬁnition good interpolations require that small changes in the source domain are reﬂected by small changes in the target domain.
We show inter-class and inter-class interpolation in Figure 5 and
Figure 6 respectively.
Particularly, we are interested in two comparing three rows of interpolations:
Transfer MNIST → MNIST → Fashion MNIST
MNIST → SC09 → MNIST Fashion MNIST →
[Table] SC
MNIST Accuracy [Table] (1) the interpolation in the source domain’s latent space, which acts a baseline for smoothness of interpolation for a pre-trained generative model, (2) transfer ﬁxed points to the target domain’s latent space and interpolate in that space, and (3) transfer all points of the source interpolation to the target domain’s latent space, which shows how the transferring warps the latent space.
We use spherical interpolation (e.g., √ pv 1 + (cid:112) (1 − p ) v 2 ) 3 since we are interpolating in the Gaussian latent space.
Note in Figure 5 that the second and third rows have comparably smooth trajectories, reﬂecting that locality has been preserved.
For inter-class interpolation in Figure 5 interpolation is smooth within a class, but between classes the second row blurs pixels to create blurry combinations of digits, while the full transformation in the third row makes sudden transitions between classes. This is expected from our training procedure as the bridging autoencoder is modeling the marginal posterior of each latent space, and thus always stays on the manifold of the actual data during interpolation.
3. D ATA E FFICIENCY AND A BLATION
Since our method is a semi-supervised method we want to know how effectively our method leverages the labeled data.
In Table 3 we show for the MNIST
→
MNIST setting the performance mea- sured by transfer accuracy with respect to the number of labeled data points.
Labels are distributed evenly among classes. The accuracy of transformations grows monotonically with the number of labels and reaches over 50% with as few as 10 labels per a class. Without labels, we also observe accuracies greater than chance due to unsupervised alignment introduced by the SWD penalty in the shared latent space.
[Table]
Finally, we perform an ablation study to conﬁrm the beneﬁts of each architecture component to transfer accuracy. For consistency, we stick to the MNIST →
MNIST setting with fully labeled data. In Table 4, we see that the largest contribution to performance is the giving the bridging VAE a domain conditioning signal, allowing it to share weights between domains, but also adapt to the speciﬁc structure of each domain. Further, the increased overlap in the shared latent space induced by the SWD penalty is reﬂected in the greater transfer accuracies.
[Table]
The training time for the conditional VAE on shared space is signiﬁcantly shorter than that of preparing pre-trained models.
For example, for MNIST ↔ MNIST our methods produce high-quality results within several minutes of training while pre-trained model takes hours.
The comparison is more observable for SC09’s case where our method takes half an hour while the pre-trained model takes days to train.
R ELATED W ORK
We discuss two aspects of existing research focuses that are related to our work, followed by how our work differentiates itself from them in order to deal with the challenge identiﬁed in this paper.
Latent Generative Models: Deep latent generative models are usually constructed to transfer a simple, tractable distribution p ( z ) into the approximation of population distribution p ∗ ( x ) , through an expressive neural network function.
Such models include VAE (Kingma & Welling, 2013) and
GAN (Goodfellow et al., 2014).
GANs are trained with an accompany classiﬁer that attempts to distinguish between samples from the decoder and the true dataset.
VAEs, in contrast, are trained with an encoder distribution q
( z | x ) as an approximation to the posterior p ( z | x ) using variational approximation through the use of evidence lower bound (ELBO).
These classes of models have been thoroughly investigated in many applications and variants (Gulrajani et al., 2017; Li et al., 2017; Bikowski et al., 2018) including conditional generation (Mirza & Osindero, 2014), generation of one domain conditioned on another (Dai et al., 2017;
Reed et al., 2016), generation of high-quality images (Karras et al., 2018) and long-range structure in music (Roberts et al., 2018).
In terms of overall VAE structure, Zhao et al.
(2017) studies options to build hierarchical VAEs.
Domain Transfer: The domain transfer enables transfer between images (Isola et al., 2016; Zhu et al., 2017; Li et al., 2018; Li, 2018), audio (Mor et al., 2018) and video (Wang et al., 2018), intuitively mapping between two domains where the variations in one domain should be reﬂected in the other.
Besides visually appealing results, domain transfer also enables application such as image colorization Zhang et al.
Domain transfer is also proposed to be done through jointly training of generative models (Lu, 2018).
Also, the behavior of domain transfer models also attracts attention.
For example, Chu et al.
(2017) suggests that image transfer does only local, texture level transfer.
To enable transfer between possibly drastically different domains, Our work proposes to use VAE in modeling the latent space of pre-trained generative models, in several aspects differentiating itself from related work. Generally, the modeling of latent space is different from modeling directly the data domains as most of latent generative models naturally do, also, the transfer in a more abstract semantics and between heterogeneous domains differs from most domain transfer method which focusing on locality and similar domains. More speciﬁcally, regarding modeling latent spaces, Zhao et al. (2017) suggests training a hierarchical VAE on a single domain should be done end-to-end, whose extension to multiple domains seems non-trivial and likely to suffer from data efﬁcient issues.
Instead, our proposed work, though enabling separation of pre-trained model and conditional shared VAE, apply to domain transfer setting while overcoming this shortcoming.
Moreover, regarding shared latent space, Liu et al.
(2017) proposes to use shared latent space for two generative models on data domains.
It requires joint training of generative models on both domains using dense supervision which is infeasible for drastically different domains.
Our work that leverages pre-trained generative model and model the latent space instead of data domains addresses to this limitation.
A PPENDIX A T RAINING T ARGET D ESIGN
We want to archive following three goals for the proposed VAE for latent spaces:
It should be able to model the latent space of both domains, including modeling local changes as well.
It should encode two latent spaces in a way to enable domain transferability.
This means encoded z 1 and z 2 in the shared latent space should occupy overlapped spaces.
The transfer should be kept in the same class.
That means, regardless of domain, z s for the same class should occupy overlapped spaces.
With these goals in mind, we propose to use an optimization target composing of three kinds of losses.
In the following text for notational convenience, we denote approximated posterior Z (cid:48)
d (cid:44) q
( z (cid:48) | z d , D = d ) , z d ∼ q ( z d | x d ) , x d ∼ p ( x d ) for d ∈ { 1 , 2 } , the process of sampling z (cid:48)
d from domain d .
Modeling two latent spaces with local changes.
VAEs are often used to model data with local changes in mind, usually demonstrated with smooth interpolation, and we believe this property also applies when modeling the latent space of data.
Consider for each domain d ∈ { 1 , 2 } , the VAE is ﬁt to data to maximize the ELBO (Evidence Lower Bound)
[Equation]
where both q and g are ﬁt to maximize L ELBO d .
Notably, the latent space z s are continuous, so we choose the likelihood π
( z ; g ) to be the product of N ( z ; g, σ 2 I ) , where we set σ to be a constant that effectively sets log π
( z ; g ) =
||
z − g || 2 , which is the L2 loss in Figure 1 (a).
Also, D KL is denoted as KL loss in Figure 1 (a).
Cross-domain overlapping in shared latent space.
Formally, we propose to measure the cross-domain overlapping through the distance between following two distributions as a proxy: the distribution of z (cid:48) from source domain (e.g., z (cid:48) 1 ∼ Z (cid:48) 1 ) and that from the target domain (e.g., z (cid:48) 2 ∼ Z (cid:48) 1 ).
We use Wasserstein Distance (Arjovsky et al., 2017) to measure the distance of two sets of samples (this notion straightforwardly applies to the mini-batch setting) S (cid:48) 1 and S (cid:48) 2 , where S (cid:48) 1 is sampled from the source domain z (cid:48) 1 ∼ Z (cid:48) 1 and S (cid:48) 1 from the target domain z (cid:48) 2 ∼ Z (cid:48) d .
For computational efﬁciency and inspired by Deshpande et al.
(2018), we use SWD, or Sliced Wasserstein Distance (Bonneel et al., 2015) between S (cid:48) 1 and S (cid:48) 2 as a loss term to encourage cross-domain overlapping in shared latent space.
This means in practice we introduce the loss term
[Equation]
where Ω is a set of random unit vectors, proj( A, a ) is the projection of A on vector a , and W 22 ( A, B ) is the quadratic Wasserstein distance, which in the one-dimensional case can be easily solved by monotonically pairing points in A and B , as proven in Deshpande et al.
Intra-class overlapping in shared latent space.
We want that regardless of domain, z s for the same class should occupy overlapped spaces, so that instance of a particular class should retain its label through the transferring.
We therefore introduce the following loss term for both domain d ∈ { 1 , 2 }
[Equation]
where H is the cross entropy loss, f ( z (cid:48) ) is a one-layer linear classiﬁer, and l x (cid:48) is the one-hot representation of label of x (cid:48) where x (cid:48) is the data associated with z (cid:48) .
We intentionally make classiﬁer f as simple as possible in order to encourage more capacity in the VAE instead of the classiﬁer.
Notably, unlike previous two categories of losses that are unsupervised, this loss requires labels and is thus supervised.
In Figure 7 we show the intuition to design and the contribution to performance from each loss terms.
[Figure]
scenario enables transfer class preservation changes highlighted in Figure overall observation is that each proposed component contributes positively to the performance in this synthetic data, which serves as a motivation for our decision to include all of them.
A PPENDIX B M ODEL A RCHITECTURE
The model architecture of our proposed VAE is illustrated in Figure B.
The model relies on Gated
Mixing Layers, or GML.
We ﬁnd empirically that GML improves performance by a large margin than linear layers, for which we hypothesize that this is because both the latent space ( z 1 , z 2 ) and the shared latent space z (cid:48) are Gaussian space, GML helps optimization by starting with a good initialization.
We also explore other popular network components such as residual network and batch normalization, but ﬁnd that they are not providing performance improvements.
Also, the condition is fed to encoder and decoder as a 2-length one hot vector indicating one of two domains.
For all settings, we use the dimension of shared latent space 100 , β SWD = 1 . 0 and β CLs = 0 .
Speciﬁcally, for MNIST ↔ MNIST and MNIST ↔ Fashion MNIST, we use the dimension of shared latent space 8 , 4 layers of FC (fully connected layers) of size 512 with ReLU, β KL = 0 .
05 , β SWD = 1 . 0 and β CLs = 0 .
05 ; while for MNIST ↔ SC09, we use the dimension of shared latent space 16 , 8 layers of FC (fully connected layers) of size 1024 with ReLU β KL = 0 .
01 , β SWD = 3 . 0 and β CLs = 0 .
The difference is due to that GAN does not provide posterior, so the latent space points estimated by the classiﬁer is much harder to model.
For optimization, we use Adam optimizer(Kingma & Ba, 2014) with learning rate 0 .
001 , beta 1 = 0 .
9 and beta 2 = 0 .
We train 50000 batches with batch size 128 .
We do not employ any other tricks for VAE training.
sigmoid gates dZ
Z = (1-gates) x Z’ + gates x dZ
Linear (a) Gated Mixing Layer
Linear d=1 z
GML
Concat
FC ReLU Layers
GML softplus z' μ σ d=
Concat
FC ReLU Layers
GML z(b)
Conditional VAE
A PPENDIX C S UPPLEMENTARY F IGURES
MNIST Digits
[Table]
[Figure]
[Table]
This mapping is made according to Labels information available at https zalandoresearch