L EARNING D ISENTANGLED R EPRESENTATIONS WITH R EFERENCE -B ASED V ARIATIONAL A UTOENCODERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning disentangled representations from visual data, where different high-level generative factors are independently encoded, is of importance for many computer vision tasks.
Supervised approaches, however, require a signiﬁcant annotation effort in order to label the factors of interest in a training set.
To alleviate the annotation cost, we introduce a learning setting which we refer to as “referencebased disentangling”.
Given a pool of unlabelled images, the goal is to learn a representation where a set of target factors are disentangled from others.
The only supervision comes from an auxiliary “reference set” that contains images where the factors of interest are constant.
In order to address this problem, we propose reference-based variational autoencoders, a novel deep generative model designed to exploit the weak supervisory signal provided by the reference set.
During training, we use the variational inference framework where adversarial learning is used to minimize the objective function.
By addressing tasks such as feature learning, conditional image generation or attribute transfer, we validate the ability of the proposed model to learn disentangled representations from minimal supervision.
I NTRODUCTION
Natural images can be considered the result of a generative process involving many factors of variation.
For instance, the appearance of a face is determined by the interaction between many latent variables including the pose, the illumination, the subject’s age and expression.
Given that the interaction between these underlying explanatory factors is usually very complex, inverting the generative process is extremely challenging.
From this perspective, learning disentangled representations where different high-level generative factors are independently encoded, can be considered one of the most relevant problems in computer vision (Bengio et al., 2013).
For instance, these representations can be applied to complex classiﬁcation tasks given that features correlated with the image labels can be identiﬁed.
We can ﬁnd another example in conditional image generation (van den Oord et al., 2016; Yan et al., 2016), where disentangled representations allow to manipulate desired attributes in synthesized images.
Motivation: Latent variable models have a long history as tools to learn abstract data representations.
The main idea behind these methods is to deﬁne a probabilistic model relating the observations, e.g . images, with a set of latent variables encoding the factors of variation underlying the data.
In recent years, Variational autoencoders (VAEs) (Kingma & Welling, 2014) have emerged as a powerful latent variable model coupling deep learning with variational inference.
However, even though
VAEs are able to build generic data representations, they are trained in an unsupervised manner and, thus, they lack a mechanism to impose speciﬁc high-level semantics on the latent variables.
In order to address this limitation, different semi-supervised autoencoders have been proposed (Rifai et al., 2012; Kingma et al., 2014).
In these approaches, a subset of the latent factors are assumed to be manually labelled in a training set.
These annotations provide supervision to the model, and allow to disentangle the labelled variables from the remaining generative factors.
The main drawback of this strategy is that it may require a signiﬁcant annotation effort depending on the problem.
For instance, if we are interested in disentangling factors related to detailed facial expressions, we need to annotate images according to a large number of facial muscle activations with their corresponding intensities (Ekman & Rosenberg, 1997).
[Figure]
In this context, we are motivated by the following question: “Is it possible to disentangle speciﬁc factors of variation with minimal supervision?”.
Contributions.
In this paper, we introduce “reference-based disentangling”.
A learning setting in which, given a training set of unlabelled images, the goal is to learn a representation where a speciﬁc set of generative factors are disentangled from the rest.
For that purpose, supervision comes in the form of an auxiliary “reference set” containing images where the factors of interest are constant.
See Fig.
1 for illustrative examples.
Different from a semi-supervised scenario, explicit labels are not available during training.
In contrast, reference-based disentangling is a weakly-supervised task, where the reference set only provides implicit information about the target generative factors that we aim to disentangle.
The main contributions of our paper are summarized as follows:
• We propose reference-based variational autoencoders (Rb-VAEs).
Different from standard unsupervised VAEs, our model is able to impose high-level semantics into the latent variables by exploiting the weak supervision provided by the reference set.
• We identify critical limitations of the VAE objective function when used to train our model.
For this reason, we propose an alternative training procedure based on recently introduced ideas in the context of variational inference and adversarial learning.
• By learning disentangled representations from minimal supervision, we show how the proposed framework is able to naturally address different tasks such as feature learning, conditional image generation, and attribute transfer.
R ELATED WORK
Deep Generative Models have been extensively explored to model visual and other types of data during recent years.
Variational autoencoders (Kingma & Welling, 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014) have emerged as two of the most effective frameworks.
VAEs use variational inference in order to learn an encoder network that maps images to a posterior distribution over latent variables.
Similarly, a decoder network is learned that produces the conditional distribution on images given the latent variables.
GANs are also composed of two differentiable networks.
The generator network synthesizes images from latent variables, similar to the
VAE decoder.
The discriminator network discriminates between real training images and generated synthetic images.
During training, GANs employ an adversarial learning procedure which allows to simultaneously optimize the discriminator and generator parameters.
Even though GANs have been shown to generate more realistic samples than VAEs, their main limitation is the lack of an inference mechanism able to map images into their corresponding latent variables.
In order to address this drawback, there have been several attempts to combine ideas from VAEs and GANs (Larsen et al., 2015; Dumoulin et al., 2017; Donahue et al., 2017).
Interestingly, it has been shown that adversarial learning can be used to minimize the variational objective function of VAEs (Makhzani et al., 2016; Husz´ar, 2017).
Inspired by this observation, various methods such as adversarial variational Bayes (Mescheder et al., 2017), α -GAN (Rosca et al., 2017), and symmetric-VAE (sVAE)
(Pu et al., 2018) have incorporated adversarial learning into the VAE framework.
Different from this prior work, our Rb-VAE model is a deep generative model speciﬁcally designed to solve the reference-based disentangling problem.
During training, adversarial learning is used in order to minimize a variational objective function inspired by the one employed in sVAE (Pu et al., 2018).
Although sVAE was originally motivated by the limitations of the maximum likelihood cri- terion used in unsupervised VAE, we show how the variational formulation of sVAE offers speciﬁc advantages in the context of our proposed model.
Learning disentangled representations is a long standing problem in machine learning and computer vision (Bengio et al., 2013).
In the literature, we can differentiate three main paradigms to address it: unsupervised, supervised, and weakly-supervised.
Unsupervised models are trained without speciﬁc information about the generative factors of interest (Desjardins et al., 2012; Chen et al., 2016).
To address this task, the most common approach consists in imposing different constraints on the latent representation.
For instance, unsupervised VAEs typically deﬁne the prior over the latent variables with a fully-factorized Gaussian distribution.
Given that high-level generative factors are typically independent, this prior encourage their disentanglement in different dimensions of the latent representation.
Based on this observation, different approaches such as β -VAE (Higgins et al., 2017), DIP-VAE (Kumar et al., 2018), FactorVAE Kim & Mnih (2018) or β -TCVAE (Chen et al., 2018) have explored more sophisticated regularization mechanisms over the distribution of inferred latent variables.
Although unsupervised approaches are able to identify simple explanatory variables of visual data, these methods do not allow latent variables to model speciﬁc high-level factors of variation.
A straight-forward approach to overcome this limitation is to use a fully-supervised strategy.
In this scenario, models are learned by using a training set where target factors of interest are explicitly labelled.
Following this paradigm, we can ﬁnd different semi-supervised (Kingma et al., 2014;
Narayanaswamy et al., 2017), and conditional (Yan et al., 2016; Pu et al., 2016) variants of autoencoders.
In spite of the effectiveness of supervised approaches in different applications (Ma et al., 2017; Pu et al., 2016; Tran et al., 2017), obtaining explicit labels is not feasible in scenarios where we aim to disentangle a large number of factors or their annotation is difﬁcult.
An intermediate solution between unsupervised and fully-supervised methods are weaklysupervised approaches.
In this case, only implicit information about factors of variation is provided during training.
Several works have explored this strategy by using different forms of weaksupervision such as: temporal coherence in sequential data (Hsu et al., 2017; Denton et al., 2017;
Villegas et al., 2017), pairing relations between images sharing the same generative factors (Mathieu et al., 2016; Donahue et al., 2018), grouping information for training samples (Bouchacourt et al., 2017), or partial knowledge about the rendering process in computer graphics (Yang et al., 2015; Kulkarni et al., 2015).
Other than previous approaches relying on other forms of weak supervision, our proposed method addresses the reference-based disentangling problem.
In this scenario, the challenge is to exploit the implicit information provided by a training set of images where the generative factors of interest are constant, e.g . a set of faces with neutral expression.
Note that for this problem, supervised approaches are not a directly applicable since explicit labels are not available.
P RELIMINARIES : V ARIATIONAL AUTOENCODERS
Variational autoencoders (VAEs) (Kingma & Welling, 2014) are generative models deﬁning a joint distribution p θ ( x , z )
= p θ ( x | z ) p ( z ) , where x is an observation, e.g . an image, and z is a latent variable with a simple prior p ( z ) , e.g . a Gaussian with zero mean and identity covariance matrix.
Moreover, p θ ( x | z ) is typically modeled as a factored Gaussian, whose mean and diagonal covariance matrix are given by a non-linear function of z , implemented as a deep (convolutional) neural network.
The latter network is referred to as the “decoder”, or “generator”.
Given a training set of i.i.d.
samples from an unknown data distribution p ( x ) , the goal of VAEs is two-fold: (i) To learn the optimal generator parameters θ so that is possible to synthesize samples following p ( x ) from the prior p ( z ) , (ii) :
To approximate the intractable posterior p θ ( z | x ) in order to infer latent variables z from a given observation x .
For this purpose, VAEs deﬁne a variational distribution q ψ ( x , z ) =
q ψ ( z | x ) p ( x ) .
The approximate posterior q ψ ( z | x ) is deﬁned as another factored Gaussian, whose mean and diagonal covariance matrix are given as the output of an “encoder” or “inference” network with parameters ψ and taking x as an input.
During training, the parameters of the decoder and encoder are learned jointly by solving the following optimization problem:
[Equation]
which is equivalent to the minimization of the KL divergence between q ψ ( x , z ) and p θ ( x , z ) .
Note that the ﬁrst term can be interpreted as a regularization mechanism encouraging the distribution q ψ ( z | x ) to be similar to the prior p ( z ) .
The second term is known as the reconstruction error, measuring the negative log-likelihood of a generated sample x from its inferred latent variables q ψ ( z | x ) .
The described minimization problem is usually solved by using stochastic gradient descent (SGD) where p ( x ) is approximated by the training set.
The “re-parametrization trick” (Rezende et al., 2014) is used to enable gradient back-propagation across samples from q φ ( z | x ) .
R EFERENCE - BASED DISENTANGLED REPRESENTATIONS
Consider a training set of unlabelled images ( e.g . human faces) x ∈ R W × H × 3 sampled from a given distribution p u ( x ) .
Our goal is to learn a latent variable model deﬁning a joint distribution over x and latent variables e ∈ R D e and z ∈ R D z .
Whereas e is expected to encode information about a set of generative factors of interest, e.g . facial expressions, z should model the remaining factors of variation underlying the images, e.g . pose, illumination, age, etc.
From now on, we will refer to e and z as the “target” and “common factors”, respectively.
In order to disentangle them, we are provided with an additional set of reference images sampled from p r ( x ) , representing a distribution over x where target factors e are constant e.g . neutral faces.
Given p r ( x ) and p u ( x ) , we deﬁne an auxiliary binary variable y ∈ { 0 , 1 } indicating whether an image x has been sampled from the unlabelled or reference distributions, i.e . p ( x | y = 0) =
p u ( x ) and p ( x | y = 1) =
p r ( x ) .
In reference-based disentangling we aim to exploit the weak-supervision provided by y in order to effectively disentangle target and common factors.
4. R EFERENCE - BASED VARIATIONAL AUTOENCODERS
In this section, we present our reference-based variational autoencoder (Rb-VAE): a deep latent variable model that deﬁnes a joint distribution p θ ( x , z , e , y ) as
[Equation]
where conditional dependencies are designed to explicitly address the reference-based disentangling problem.
See Figure 2 for a schematic illustration of our model.
We deﬁne p θ ( x | z , e ) =
L ( x |G θ ( z , e ) , λ ) , where G θ ( z , e ) is a decoder network that maps a pair of latent variables ( z , e ) to an image deﬁning the mean of a Laplace distribution L with ﬁxed scale parameter λ .
Note that we use a Laplace distribution instead of the Gaussian usually employed in the VAEs.
The reason is that the negative log-likelihood is equivalent to the (cid:96) 1 -loss which is known to encourage sharper image reconstructions with better visual quality.
To reﬂect the assumption of constant target factors across reference images, we deﬁne the conditional distribution over e given y = 1 as a delta peak centered on a learned vector e r ∈ R D e , i.e . p ( e | y = 1)
=
δ ( e − e r ) .
In contrast, for y = 0 , the conditional distribution is set to a unit Gaussian: p ( e | y = 0)
=
N ( e | 0 , I ) .
In the following, we denote p ( e | y = 0)
=
p ( e ) .
Contrary to the case of target factors e , the prior over common factors z is equal for reference and unlabelled images, and taken to be a unit Gaussian y for p ( z ) =
N ( z | 0 , I ) .
Finally, we assume a uniform prior over y , i.e . p ( y = 0)
=
p ( y = 1) = 12 .
4. C ONVENTIONAL VARIATIONAL LEARNING
Following the VAE framework described in Section 3, we can deﬁne a variational distribution q ψ ( z , e , x , y ) =
q ψ ( z | x ) q ψ ( e | x , y ) p ( x , y ) , and learn the model parameters θ by minimizing the KL divergence between q ψ and p θ .
In this case, note that the conditionals q ψ ( e | x , y ) and q ψ ( z | x ) provide a factored approximation of the intractable posterior p θ ( e , z | x , y ) , allowing to infer target and common factors e and z given an image x .
Given a reference image, i.e . with y = 1 , the target factors q ψ ( e | x , y = 1) are known to be equal to the reference value e r .
On the other hand, given an non-reference image, i.e . with y = 0 , we deﬁne the approximate posterior q ψ ( e | x , y = 0)
=
N ( e |E µ ( x ) , E σ ( x )) , where the means and diagonal covariance matrices of a conditional Gaussian distribution are given by non-linear functions E µ ( x ) and E σ ( x ) .
Similarly, we use an additional network to model q ψ ( z | x ) =
N ( z |Z µ ( x ) , Z σ ( x )) .
e z x
U n l a b e ll e d
R e f e r e n c
e e z y e z x y e
z x z x e r y (a) (c) (d)
y e z x e r e r
e z x z (b) e r
Optimization with SGVB: In Appendix (A) we show that the minimization of the KL divergence between q ψ and p θ can be expressed as:
[Equation]
(3) where the second and fourth terms of the expression correspond to the reconstruction errors for unlabelled and reference images respectively.
Note that for reference images, no inference over target factors e is needed.
Instead, the generator reconstructs them using the optimized parameter e r .
Similar to standard VAEs, the remaining terms consist of KL divergences between approximate posteriors and priors over the latent variables.
The minimization problem deﬁned in Eq.
(3) can be solved using SGD and the re-parametrization trick in order to back-propagate the gradient when sampling from q ψ ( e | x ) and q ψ ( z | x ) .
4. V ARIATIONAL L EARNING WITH S YMMETRIC KL DIVERGENCE
The main limitation of the variational objective deﬁned in Eq.
(3) is that it does not guarantee that common and target factors will be effectively disentangled in z and e respectively.
In order to understand this phenomenon, it is necessary to analyze the role of the conditional prior p ( e | y ) in
Rb-VAEs.
By deﬁning p ( e | y = 1) as a delta function, the model is forced to encode into z all the generative factors of reference images, given that they must be reconstructed via p θ ( x | z , e r ) with constant e r .
Therefore, p ( e | y ) is implicitly encouraging q ψ ( z | x ) to encode common factors present in reference and unlabelled samples.
However, this mechanism does not avoid the scenario where target factors are also encoded into latent variables z .
More formally, the minimization of (3) does not prevent a degenerate solution p θ ( x | z , e ) =
p θ ( x | z ) , where the inferred latent variables by q ψ ( e | x ) are uninformative and the decoder ignores them.
In order to address this limitation, we propose to optimize an alternative variational expression inspired by unsupervised Symmetric VAEs (Pu et al., 2018).
Speciﬁcally, we add into the minimized objective the reversed KL between q ψ and p θ as:
[Equation]
In order to understand why this additional term allows to mitigate the degenerate solution p θ ( x | z , e ) =
p θ ( x | z ) , it is necessary to observe that its minimization is equivalent to:
[Equation]
see Appendix A for a detailed derivation.
In the deﬁned expression, the two KL divergences encourage images generated using p ( z ) , p ( e ) and e r to be similar to samples from the real distributions p r ( x ) and p u ( x ) .
On the other hand, the remaining terms correspond to reconstruction errors over latent variables z , e inferred from generated images drawn from p θ .
As a consequence, the minimization of these errors is encouraging the decoder p θ ( x | z , e ) to generate images x by taking into account latent variables e , since the latter must be reconstructed via q ψ ( e | x ) .
In conclusion, the addition of the reversed KL in the objective avoids the degenerate solution that ignores variables e .
Optimization via Adversarial Learning.
Given the introduction of the reversed KL divergence, the learning procedure described in Section 4.2 can not be directly applied to the minimization of
Eq.
However, note that the objective function deﬁned in Eq.
(4) is equivalent to:
[Equation]
corresponding to the minimization of
[Equation]
A difference of two expectations over the log-density ratio of q ψ ( e , z | x ) p u ( x ) and p θ ( x | e , z ) p ( z ) p ( e ) .
(ii)
An analogous expression for distributions q ψ ( z | x ) p r ( x ) and p θ ( x | e r , z ) p ( z )) .
By approximating both density ratios with two auxiliary functions:
[Equation]
we can express the minimization problem in Eq.
(6) as:
[Equation]
which can be solved with SGD.
Concretely, we can evaluate functions d ξ ( x , z , e ) , d γ ( x , z ) and back-propagate the gradients w.r.t parameters ψ and θ by using the re-parametrization trick over samples of x , e and z .
The main challenge of approximating the log-density ratio using the functions deﬁned in Eq.
(7), is that parameters ξ and γ are not known and must be also optimized.
Fortunately, the log-ratio between two distributions can be estimated by using logistic regression (Bickel et al., 2009).
For instance, given ﬁxed encoder and generator parameters θ and ψ , we can optimize ξ by solving:
[Equation]
where σ ( · ) refers to the sigmoid function.
A similar strategy can be applied to optimize parameters of d γ ( x , z ) .
This approach is analogous to adversarial unsupervised methods such as ALI (Dumoulin et al., 2017), where the function d γ ( · ) acts as a discriminator trying to distinguish whether pairs of reference images x and latent variables z generated by q ψ and p θ .
However, in our case we have an additional discriminator d ξ operating over unlabelled images and its corresponding latent variables z and e .
Interestingly, note that d ξ encourages the distribution q ψ ( z , e | x ) to be similar to the prior p ( z , e ) .
Therefore, it forces representations z and e to be conditionally independent.
This is especially interesting in our context since we assume that target and common factors are uncorrelated.
To conclude, it is also interesting to observe that the discriminator d γ ( x , z ) is implicitly encouraging latent variables z to encode only information about the common factors. The reason is that samples generated from p θ ( x | z , e r ) p ( z ) are forced to be similar to reference images. As a consequence, z can not contain information about target factors, which must be encoded into e. Using previous deﬁnitions, we use an adversarial procedure where model and discriminators parameters ( θ , ψ ), and ( ξ , γ ) are simultaneously optimized by minimizing and maximizing Eqs.
(8) and (9) respectively.
The algorithm used to process one batch during SGD is shown in Appendix B. In
Rb-VAEs, the discriminators d γ ( · ) and d ξ ( · ) are also implemented as deep convolutional networks.
Explicit log-likelihood maximization.
As shown in Eqs.
(3) and (5), the minimization of the symmetric KL divergence encourages low reconstruction errors for images and inferred latent variables.
However, by using the proposed adversarial learning procedure, the minimization of these terms becomes implicit.
As shown in (Dumoulin et al., 2017; Donahue et al., 2017), this can cause original samples to differ substantially from their corresponding reconstructions.
In order to address this drawback, we use a similar strategy as (Pu et al., 2018; Li et al., 2017), and add the reconstruction terms:
[Equation]
into the learning objective, minimizing them together with Eq.
In preliminary experiments, we found that the explicit addition of these reconstructions terms during learning is important to achieve low reconstruction errors and increase stability during the adversarial training procedure.
Figurein the Appendix illustrates all the different losses minimized during learning in sRBD-VAE.
E XPERIMENTS 5. D ATASETS
To validate our approach and to compare to existing work, we consider two different reference-based disentangling problems.
Digit style disentangling: The goal is to model style variations from hand-written digits.
Concretely, we consider the digit style as a set of three different properties: scale, width and color.
In order to address this task from a reference-based perspective, we consider half of the original training images in the MNIST dataset (LeCun et al., 1998) as our reference distribution (30k examples).
The unlabelled set is synthetically generated by applying different transformations over the remaining half of images: (1) Simulation of stroke widths by using a dilation with a given ﬁlter size; (2) Digit colorization by multiplying the RGB components of each pixel by a 3D RGB color vector; (3) Size variations by down-scaling the image by a given factor and applying zero-padding to recover the original resolution.
We randomly transform each image twice to obtain a total of 60k unsuperivsed images.
More details can be found in Appendix C.
Facial expression disentangling: We consider disentangling of facial expressions by using a reference set of neutral faces.
For the unlabelled dataset we use a subset of the AffectNet dataset (Mollahosseini et al., 2017), which contains a large quantity of facial images.
This dataset is especially challenging since images were collected “in the wild” and exhibit a large variety of natural expressions.
A subset of the images are annotated according to different basic facial expressions: happiness, sadness, surprise, fear, disgust, anger , and contempt .
We use these labels for quantitative evaluation.
Given that we found that many neutral images in the original database were not correctly annotated, we collected a separate reference set, see Appendix C for details.
The unlabelled and reference sets consist of 150k and 10k images, respectively.
5. E XPERIMENTAL SETUP
Baselines: We compare the two different variants of our proposed method: Rb-VAE, trained using the standard variational objective (Section 4.2), and sRb-VAE, learned by minimizing the symmetric KL divergence (Section 4.3).
To demonstrate the advantages of exploiting the weak-supervision provided by reference images, we compare both methods with various state-of-the-art unsupervised approaches based on the VAE framework: β -VAE (Higgins et al., 2017), β -TCVAE
(Chen et al., 2018)
sVAE
(Pu et al., 2018), DIP-VAE-I and DIP-VAE-II (Kumar et al., 2018).
Note that β -VAE DIP-VAE and have β -TCVAE been speciﬁcally proposed for learning disentangled representations, showing better performance than other unsupervised methods such as InfoGAN (Chen et al., 2016).
On the other hand, sVAE is trained using a similar variational objective as sRb-VAE, and can therefore be considered an unsupervised version of our method.
We also use standard VAE (Kingma & Welling, 2014) as an additional baseline.
Finally, in order to evaluate an alternative approach able to exploit the reference-set, we have implemented (Mathieu et al., 2016) adapting it to our context.
Note that the amount of supervision used in the cited method is larger than the available in referencebased disentangling. Concretely, we do not have information about unlabelled samples sharing the same target factors (e.g the same expression). In contrast, this information is only available for reference images. This fact renders the original learning algorithm proposed in (Mathieu et al., 2016)
[Table]
inapplicable.
However, we have modiﬁed it order to use only information from reference images by removing the reconstruction losses assuming known pairs for unlabelled samples.
Implementation details: The different components of our method are implemented as deep neural networks.
For this purpose, we have used an architecture based on the main building blocks employed by (Karras et al., 2018).
More concretely, generator networks are implemented as a sequence of convolutions, Leaky-ReLU non-linearities, and nearest-neighbour up-sampling operations.
Encoders and discriminators follow a similar architecture, using average pooling for down-sampling.
Channel normalization is used for generators and encoders.
See Appendix D for more details on the precise architecture.
Our code will be made available online upon publication.
For a fair comparison, we have developed our own implementation for all the evaluated methods in order to use the same network architectures and hyper-parameters.
During optimization, SGD is employed using the
Adam optimizer (Kingma & Ba, 2015) (with α = 10 − 4 , β 1 = 0 .
5 , β 2 = 0 .
99 , (cid:15) =
10 − 8 ), and a batch size of 36 images.
For the MNIST and AffectNet databases, the models are learned for 30 and 20 epochs respectively.
The number of latent variables for the encoders has been set to 32 for all the experiments and models.
The λ parameter in the Laplace distribution is set to 0.01.
5. Q UANTITATIVE EVALUATION : F EATURE L EARNING
A common strategy to evaluate the quality of learned representations is to measure the amount of information that they convey about the underlying generative factors (Eastwood & Williams, 2018).
In the context of reference-based disentangling, we are interested in modelling the target factors that are constant in the reference distribution.
In this experiment, we use this observation in order to quantitatively evaluate our method.
For this purpose, we use the learned representations as feature vectors and train a low-capacity model estimating the target factors involved in each problem.
Concretely, in the MNIST dataset we employ a set of linear-regressors predicting the scale, width and color parameters for each digit.
To predict the different expression classes in the
AffectNet dataset, we use a linear classiﬁer.
For evaluation, we split each dataset in three subsets.
The ﬁrst is used to learn each generative model.
Then, the second is used for training the regressors or classiﬁer.
The third is used to evaluate the predictions in terms of the mean absolute error and per-class accuracy for the MNIST and AffectNet datasets, respectively.
In MNIST, the second and third subset have been randomly generated from the original MNIST test set using the procedure described in Section 5.1 (5k images each).
For AffectNet, we randomly select 500 images for each of the seven expressions from the original dataset, yielding 3,500 images per fold.
Table 1 shows the results obtained by the unsupervised models considered 1 and the proposed RbVAE and sRb-VAE.
For Rb-VAE and sRb-VAE, we used the inferred latent variables e as features since the distribution over these variables is expected to encode the information regarding the target factors.
For the unsupervised models we use all the latent variables.
By analyzing the results, we obtain the following conclusions.
First, the unsupervised approach DIP-VAE-I achieves better average results than Rb-VAE for MNIST.
Moreover, for facial expression recognition, β -TCVAE achieves comparable or better performance in several cases.
This may seem counter-intuitive because, unlike Rb-VAE, DIP-VAE-I is trained without the weak-supervision provided by reference
-VAE-I, DIP-VAE-II, β -VAE and β -TCVAE we tested different regularization parameters in the range [1 , 50] , and report the best results.
images. However, it conﬁrms our hypothesis that the learning objective of Rb-VAE does not explicitly encourage the disentanglement between target and common factors.
In contrast, we can see that sRb-VAE obtains comparable or better results than rest of the methods in most cases.
Moreover, it achieves the best average performance in both datasets.
This demonstrates that the information provided by the reference distribution is effectively exploited by the symmetric KL objective used to train sRb-VAE.
In order to further validate the ability of the proposed method to disentangle the target factors, we have followed the same evaluation protocol for Rb-VAE and sRb-VAE but considering the latent variables z as features.
The average performance obtained by Rb-VAE is .349 and .195 for AffectNet and MNIST respectively.
On the other hand, sRb-VAE achieves .335 and .189 .
Note that for both methods these results are signiﬁcantly worse compared to using e as a representation in Table 1.
This suggests that latent variables z are mainly modelling the common factors between reference and unlabelled images.
The qualitative results presented in the next section conﬁrm this.
To conclude, note that sRBD-VAE also obtains better performance than (Mathieu et al., 2016) in both data-sets.
So even though this method also uses reference images during training, our results conﬁrms that sRBD-VAE better exploits the weak-supervision provided in reference-based disentangling.
5. Q UALITATIVE EVALUATION
In contrast to unsupervised approaches, reference images can be used by our model in order to explicitly encode the target and common factors into two different subsets of latent variables.
This directly enables tasks such as conditional image synthesis or attribute transfer.
In this section, we illustrate the potential applications of our proposed model in this type of applications.
Conditional image synthesis.
The goal is to transform real images by modifying only the target factors e .
For instance, given a face of an individual, we aim to generate images of the same subject exhibiting different facial expressions.
For this purpose, we use our model in order to infer the common factors z .
Then, we sample a vector e ∼ N ( 0 , 1 ) and use the generator network to obtain a new image from e and z .
In Fig.
3 we show examples of samples generated by Rb-VAE and sRbVAE following this procedure.
As we can observe, sRb-VAE generates more convincing results than its non-symmetric counterpart.
In the AffectNet database, the amount of variability in Rb-VAE samples is quite low.
In contrast, sRb-VAE is able to generate more diverse expressions related with eyes, mouth and eyebrows movements.
Looking at the MNIST samples, we can draw similar conclusions.
Whereas both methods generate transformations related with the digit color, Rb-VAE does not model scale variations in e , while sRb-VAE does.
This observation is coherent with results reported in Table 1, where Rb-VAE offers a poor estimation of the scale.
Visual attribute transfer.
Here we transfer target factors e between a pair of images A and B.
For example, given two samples from the MNIST dataset, the goal is to generate a new image with the number in A modiﬁed with the style in B. Using our model, this can be easily achieved by synthesizing a new image from latent variables e and z inferred from A and B respectively.
Figureshows images generated by sRb-VAE and Rb-VAE in this scenario.
In this case, we can draw similar conclusions than the previous experiment.
Rb-VAE is not able to swap target factors related with the digit scale in the MNIST dataset, unlike sRb-VAE which better model this variation factor.
On the AffectNet images, both methods are able to keep most of the information regarding the identity of the subject, but again Rb-VAE leads to weaker expression changes than sRb-VAE.
These qualitative results demonstrate that the standard variational objective of VAE is sub-optimal to train our model, and that the symmetric KL divergence objective used in sRb-VAE allows to better disentangle the common and target factors.
C ONCLUSIONS
In this paper we have introduced reference-based disentangling, a novel weakly supervised learning problem, and proposed reference-based variational autoencoder models to address it.
We have shown that the standard variational learning objective used to train VAE can lead to degenerate solutions when it is applied in our setting, and proposed an alternative training strategy that exploits adversarial learning.
Comparing the proposed model with previous state-of-the-art unsupervised approaches, we have demonstrated its ability to learn disentangled representations from minimal supervision, and its potential application to tasks such as feature learning, conditional image generation, and attribute transfer.
A PPENDIX A M ATHEMATICAL DERIVATIONS
Equivalence between KL ( q ψ ( z , e , x , y ) (cid:107) p θ ( x , z , e , y )) and Eq.
[Equation]
We denote H r
( X ) and H u ( X ) as the entropy of the reference and unlabelled distributions p r ( x ) and p u ( x ) respectively.
Note that they can be ignored during the minimization since are constant w.r.t.
parameters θ and ψ .
For the second equality, we have used the deﬁnitions p ( x | y = 0)
=
p u ( x ) , p ( x | y = 1) =
p r ( x ) and assumed p ( y = 0) = p ( y = 1) = 12 .
Moreover, we have exploited the fact that q ψ ( e | x , y = 1) and p ( e | y = 1) are deﬁned as delta functions and, therefore,
E p ( e | y =1) log ( p ( e | y =1) q ψ ( e | y =1) )
= 0 .
We denote p ( e | y = 0)
=
p ( e ) and q ψ ( e | x , y = 0)
=
q ψ ( e | x ) for the sake of brevity.
Equivalence between KL ( p θ ( x , z , e , y ) (cid:107) q ψ
( z , e , x , y )) and the expression in Eq.
[Equation]
z ) p ( e | y ) p ( y ) log (cid:32) p θ ( x | e , z ) p ( z ) p ( e | y ) p ( y )
q ψ ( z , e , x , y ) p ( x | y ) p ( y ) (cid:33)
d x d z d e (14) =
(cid:90) x,e,z p θ ( x | e , z ) p ( z ) p ( e ) log (cid:32) p θ ( x | e , z ) p ( z ) p ( e ) q ψ ( e | x ) q ψ ( z | x ) p U ( x ) (cid:33)
d x d z d e + (cid:90) x,z p θ ( x | e r , z ) p ( z ) log (cid:32) p θ ( x | e r , z ) p ( z ) q ψ ( z | x ) p u ( x ) (cid:33) d x d z d e (15) =
2 E p ( z ) p ( e ) E p θ ( x | e , z ) (cid:34) log (cid:32) p θ ( x | e , z ) p ( x ) u (cid:33) − log( q ψ ( e | x ) q ψ ( z | x )) (cid:35) +
E p ( z ) E p θ ( x | e r , z ) (cid:34) log (cid:32) p θ ( x | e r , z ) p ( x ) r (cid:33) − log( q ψ ( z | x )) (cid:35) − H
( z ) − 1 2 H ( e ) (16) =
2 E p ( z ) p ( e ) (cid:104) KL ( p θ ( x | z , e ) (cid:107) p u ( x )) − E p θ ( x | z , e ) [log( q ψ ( z | x ))
+ log( q ψ ( e | x ))] (cid:105)
[Equation]
We have used the same deﬁnitions and assumptions previously discussed.
Moreover, we denote
H ( z ) and H ( e ) as the entropy of the priors p ( z ) and p ( e ) .
Again, we can ignore these terms when we are optimizing w.r.t parameters ψ and θ .
Equivalence between the minimization of the symmetric KL divergence in Eq.
(4) and the expression in Eq.
[Equation]
A PPENDIX B P SEUDO - CODE FOR ADVERSARIAL LEARNING PROCEDURE
Alg.
1 shows pseudo-code for the adversarial learning algorithm described in Sec.
4.3 of the paper. Fig.
5 illustrates all the different losses minimized during learning in sRBD-VAE.
[Table]
[Figure]
[Table]
[Equation]
A PPENDIX C D ATASETS
Examples of reference and unlabelled images for MNIST and AffectNet are shown in Fig.
Following, we provide more information about the used datasets
[Table]
R e f e r e n c e I m a g e s
U n l a b e ll e d
I m a g e s
Common factors ( z )
Target factors
[Figure]
e )
Target reference
[Figure]
[Figure]
C. MNIST
We use slightly modiﬁed version of the MNIST images: the size is increased to 64 × 64 pixels and an edge detection procedure is applied to keep only the boundaries of the digit.
We obtain the samples in the unlabelled dataset by applying the following transformations over the MNIST images:
Width :
Generate a random integer in the range { 1 , . .
, 10 } using a uniform distribution.
Apply a dilation operation over the image using a squared kernel with pixel-size equal to the generated number.
Color : Generate a random 3D vector c ∈ [0 , 1] 3 using a uniform distribution.
Normalize the resulting vector as ˆ c
=
c/ ||
c
|| 1 .
Multiply the RGB components of all the pixels in the image by ˆ c . 3.
Size : Generate a random number in the range [0 . , 1] using a uniform distribution.
Downscale the image by a factor equal to the generated number.
Apply zero-padding to the resulting image in order to recover the original resolution.
The data and code to generate it will be made available online upon publication.
C. A FFECT N ET
Reference set collection.
We collected a reference set of face images with neutral expression.
We applied speciﬁc queries in order to obtain a large amount of faces from image search engines.
Then, ﬁve different annotators ﬁltered them in order to keep only images showing a neutral expression.
This was motivated because we found that many neutral images in the AffectNet dataset Mollahosseini et al.
(2017) were not accurate.
As detailed in the original paper, each image was only labelled by a single subject.
As a consequence, the agreement between annotators was shown to be low for neutral images.
In contrast, in our reference-set, each image was annotated in terms of “neutral” / “non-neutral” by two different annotators.
In order to ensure a higher label quality compared to [Figure] the AffectNet, only the images where both annotators agreed were added to the reference-set.
The collected dataset will be made available upon publication.
Pre-processing.
In order to remove 2D afﬁne transformations such as scaling or in-plane rotations, we apply an alignment process to the face images.
We localize facial landmarks using (Xiong &
De la Torre, 2013).
Then, we apply Procrustes analysis in order to ﬁnd an afﬁne transformation aligning the detected landmarks with a mean shape.
Finally, we apply the transformation to the image and crop it.
The resulting image is then re-sized to a resolution of 96 × 96 pixels.
A PPENDIX D N ETWORK ARCHITECTURES
Fig.
7 illustrates the network architectures used in our experiments.
CN refers to pixel-wise normalization as described in (Karras et al., 2018).
FC deﬁnes a fully-connected layer.
For Leaky
ReLU non-linearities, we have used an slope of 0 . 2 .
Given that we normalize the images in the range [ − 1 , 1] , we use an hyperbolic tangent function as the last layer of the generator.
For the discriminator d γ ( x , z ) , we use the same architecture showed for d ξ ( x , z , e ) but removing the input corresponding to e .
In preliminary experiments, we found that the discriminator in sRb-VAE can start to ignore the inputs corresponding to latent variables e and z while focusing only on real and generated images.
In order to mitigate this problem during training, we found it effective to randomly set to zero the inputs corresponding to latent variables and images of the last fully-connected layer.
In particular, we use a probability of 0 .
A PPENDIX E A DDITIONAL RESULTS
Figures 8 and 9 show additional qualitative results for conditional image generation and visual attribute transfer, in the same spirit as the ones in 5.4.
The additional results further support the conclusions drawn in the main paper.
In Fig.
10, we also show additional images generated by sRB-VAE trained with the AffectNet dataset.
Different from the previous cases, these images have been generated by injecting random noise over both latent variables e and z .
Note that different target factors e generate similar expressions in images generated from different common factors z .
[Table]
[Figure] [Figure]