  We propose a cross-domain latent modulation mechanism within a variational
autoencoders (VAE) framework to enable improved transfer learning. Our key idea
is to procure deep representations from one data domain and use it as
perturbation to the reparameterization of the latent variable in another
domain. Specifically, deep representations of the source and target domains are
first extracted by a unified inference model and aligned by employing gradient
reversal. Second, the learned deep representations are cross-modulated to the
latent encoding of the alternate domain. The consistency between the
reconstruction from the modulated latent encoding and the generation using deep
representation samples is then enforced in order to produce inter-class
alignment in the latent space. We apply the proposed model to a number of
transfer learning tasks including unsupervised domain adaptation and
image-toimage translation. Experimental results show that our model gives
competitive performance.
