D ISENTANGLING F ACTORS OF V ARIATIONS U SING
F EW L ABELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning disentangled representations is considered a promising research direction in representation learning.
Recently, Locatello et al.
(2018) demonstrated that the unsupervised learning of disentangled representations is theoretically impossible and that state-of-the-art methods, which are often unsupervised, require access to annotated examples to select good model runs.
Yet, if we assume access to labels for model selection, it is not clear why we should not use them directly for training.
In this paper, we ﬁrst show that model selection using few labels is feasible.
Then, as a proof-of-concept, we consider a simple semi-supervised method that directly uses the labels for training.
We train more than 7000 models and empirically validate that collecting a handful of potentially noisy labels is sufﬁcient to learn disentangled representations.
I NTRODUCTION
Learning the true underlying generative model for the observed data is one of the fundamental problems of machine learning.
It is commonly accepted that the observed data x exhibits redundant features and its essence can be captured by a low dimensional latent variable z (Bengio et al., 2013;
Kulkarni et al., 2015; Chen et al., 2016; Tschannen et al., 2018).
The observations are generated by ﬁrst sampling a z from a distribution P ( z ) .
Then, x is sampled from the conditional distribution
P ( x | z ) .
The goal of disentanglement learning is to ﬁnd a transformation of the data r ( x ) which is “aligned” with the factors of variations in z .
The hope is that such representation will be useful for any downstream task (Bengio et al., 2013; Peters et al., 2017; LeCun et al., 2015; Bengio et al., 2007;
Schmidhuber, 1992; Lake et al., 2017; Goodfellow et al., 2009; Lenc & Vedaldi, 2015; Tschannen et al., 2018; Higgins et al., 2018; Suter et al., 2018).
In particular, we hope to learn a disentangled representation without supervision so that only few labels will be needed to learn any downstream task (Bengio et al., 2013;
Sch ¨ olkopf et al., 2012; Peters et al., 2017; Pearl, 2009; Spirtes et al., 1993).
Current state-of-the-art approaches enrich the Variational Autoencoders (VAEs) (Kingma & Welling, 2013) objective with some unsupervised regularizer that should encourage disentangled representations (Higgins et al., 2016;
Burgess et al., 2018; Kim & Mnih, 2018;
Chen et al., 2018; Kumar et al., 2017; Rubenstein et al., 2018).
Unfortunately, purely unsupervised disentanglement learning is theoretically impossible and unsupervised methods exhibit a considerable variance across different runs (Locatello et al., 2018).
Therefore, labels (which in this setup are the observations of the factors of variations) are in practice required to perform model selection.
In this paper, we question how much supervision is needed to successfully perform model selection on the unsupervised methods and how does it compare to a simple semi-supervised approach.
For this purpose, we re-evaluate the 12 000 trained models of Locatello et al.
(2018) using 100 andlabelled examples.
We further train more than 7000 simple semi-supervised models using the same amount of labels in a 90%/10% train/validation split.
Overall, we show that a little amount of supervision is enough to learn disentangled representations.
Our contributions can be summarized as follows: (i) we show that disentanglement metrics can be used to tune the hyperparameters of unsupervised methods even if very few labelled examples are available.
(ii) we show that adding a simple supervised loss to β -VAE outperforms unsupervised training with supervised model selection using as little as 100 labelled examples.
(iii) we show that the simple semi-supervised method we consider is robust to label noise and simpliﬁed labels.
I NCORPORATING LABEL INFORMATION
Let R u ( q φ ( z | x )) be a regularizer imposing structure on the encoder distribution in order to encourage a disentangled representation.
Then, we can group all the SoTA unsupervised approaches (Higgins et al., 2016; Burgess et al., 2018;
Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2017) under the following optimization template:
[Equation]
In practice, even though disentangled methods might train without supervision, it is assumed that some number of observations of z is available for evaluation.
Note that supervised model selection explicitely breaks the impossibility result of (Locatello et al., 2018).
While collecting a large amount of labelled data in this setup is unreasonable, we question whether labelling a very limited amount of data points is sufﬁcient to reliably obtain disentangled representations.
If we collect a large dataset of unlabelled data and a few labelled examples we have two options: either we train an unsupervised method and perform supervised model selection or we split the labels in two sets and use one training and one for validation.
As a proof-of-concept, we consider the simplest way to incorporate supervision in Equation 1:
[Equation]
where R s ( q φ ( z | x ) , z ) is some function computed only on the observed labels.
In other words, we constraint the otherwise unsupervised problem using some supervised penalty.
We now can include
R s in the optimization as a regularizer under the KKT conditions:
[Equation]
In our study, we use for R s a cross entropy loss on the ﬁrst few dimension of the encoder’s mean to align the representation to the factors of variation (which are normalized between zero and one during training and validation).
When z has more dimension than the number of factors of variation the model is misspeciﬁed and we should learn to ignore the extra dimensions.
As opposed to (Kingma et al., 2014), we directly observe the values of z and have no distinction between latent variables and class labels. (Narayanaswamy et al., 2017) introduce a semi-supervised generative model that disentangle some labelled factors of variation from the others that remain entangled. (Cheung et al., 2014), enrich the autoencoder objective with a supervised cross entropy loss that disentangle the label information (in their case the MNIST class information) from the other factors of variations (i.e. the digit’s style) that can not be disentangled.
Note that there is a rich literature on semi-supervised disentanglement, which unfortunately precedes the advent of the modern disentanglement metrics and problem formulation.
Therefore, semi-supervised methods were only evaluated qualitatively and the quantitative gap between semi-supervised methods and unsupervised ones is currently not known.
In the following section, we show that this simple model already competes with the SoTA approaches.
E XPERIMENTS
We build our experimental protocol on top of the disentanglement lib of Locatello et al.
For testing, we use the pipeline of disentanglement lib which computes the disentanglement scores on 10000 labelled points.
Setup for unsupervised methods: We use the available labels ( 100 or 1000 ) to re-evaluate the trained models of Locatello et al.
(2018) using MIG, SAP Score and DCI Disentanglement.
These results are then used to perform model selection on their trained models.
We assume we can obtain exact labels.
Note that the BetaVAE and FactorVAE metrics cannot be used for this purpose as they assume access to the generative process.
We do not use Modularity to tune hyperparameters of the unsupervised models as it was not found to be strongly correlated with the other metrics in (Locatello et al., 2018)
(see also Figure 1).
Setup for semi-supervised method: We add the supervised loss with its regularization strength to the β -VAE implementation of Locatello et al.
For the supervised regularization strength we use the same hyperparameter range of the unsupervised one.
We train more than 7000 models on the same seven datasets, with the same hyperparameters for 5 different random seeds.
In this way, we can compare directly with their > 12000 SoTA unsupervised methods.
We consider a clear train, validation and test split.
We assume to have available either 100 or 1000 labelled datapoints which we use for training and model selection (90%/10% split).
To test robustness, we consider exact labels (referred as S2-perfect ), noisy labels (the observation of each factor of variation has a 10% chance of being random, referred as S2-noisy ) and simpliﬁed labels (binned to only takevalues, referred as S2-bin ).
This last scenario is meant to simulate the process of a human labelling a realistic number of images (either 100 or 1000 ) where each factor of variation can take onlyvalues for simplicity.
During training and validation, we group the values of each factor of variation in one dimension and normalize it so it takes values in [0,1].
This simple choice can be good for continuous variables (for example for spatial location) but is likely not ideal for discrete ones (such as shape).
For model selection, we compute the supervised loss on the validation data.
[Figure] [Figure]
[Table]
Disentanglement metrics with few samples: As a ﬁrst step, we investigate whether using supervision to tune the hyperparameters of unsupervised methods is possible in the setting where only few labelled points are available.
We re-evaluated the trained methods of (Locatello et al., 2018) with 100 and 1000 samples instead of 10 000 on a different set of datapoints.
In Figure 1, we show the rank correlation of the MIG, DCI Disentanglement and
SAP score on dSprites we use for validation compared to the test disentanglement scores.
We notice a strong and consistent correlation.
The only exception seem to be the SAP score with 100 samples that does not correlate strongly with any metric.
These trends are consistent across different datasets.
From this comparison we conclude that it is possible to use a few labels to tune unsupervised models.
Therefore, we argue that it is fair to compare model selection of unsupervised methods with semi-supervised approaches.
Does supervision improve training?
In this section, we question whether we should keep all the labels to tune the hyperparameters or we should use some for training.
For each dataset and metric, we take the β -VAE setup from Locatello et al.
and compare their trained models with the same models trained with the additional supervised loss (only differences are seeds and supervised loss).
Given the large amount of comparisons, we report how often does supervision improves upon the corresponding unsupervised method for each dataset and metric.
In Table 1, we see that overall adding some supervision improves the unsupervised β -VAE, even if the supervision is noisy or not detailed (we use only 90% of the labels for training).
Note that the result of Table 1 only tells us that good models exist, but not how to ﬁnd them.
Table 2, shows the comparison after model selection.
For robust comparison, we group the models in 5 bins based on the seed and perform the selection over each bin, for each dataset and for each metric.
From this experiment, we conclude that including labels during training appears to give better disentanglement than only performing supervised selection.
In Table 3, we show that using only 100 labels our semi-supervised β -VAE is competitive with all the other SoTA unsupervised models with supervised model selection.
With 1000 labelled examples the gap is bigger as depicted in Table 5 in the appendix.
The median score of the models tuned with the
MIG with 100 labels versus our semi-supervised model can be seen in Tables 6-12 in the appendix.
Results for all the metrics and datasets with conﬁdence intervals can be seen in Figures 5-10.
In
Figure 11 we can see an example of the semi-supervised model perfectly disentangling Shapes3D.
[Table] [Table]
The semi-supervised method is most often the best one.
Disentanglement is abbreviated to DCI to
Overall, we conclude that the gain of using labels while training is higher with 1000 labels rather than 100 .
This implies that supervised model selection of the unsupervised models might be more useful when even fewer labels are available.
On the other hand, if we have access to a considerable amount of labels, it seems better to use them during training.
[Figure]
strategies.
Models are abbreviated as (0=S2-perfect 100, 1=S2-perfect 1000, 2=S2-bin 100, 3=S2-bin 1000, 4=S2noisy 100, 5=S2-noisy 1000, 6=VAE)
Noisy labels: Finally, we test how sensitive our semisupervised method is to noisy or simpliﬁed labels.
In Figure 2, we can see a standard behaviour where the variance signiﬁcantly decreases when we add more labels.
Collecting perfect labels seem to be the best strategy, but label noise and the simpliﬁed labels do not signiﬁcantly harm the performances.
The full result can be seen in Figurein the Appendix.
From this experiment, we conclude that it might be beneﬁcial to collect some labels for training even if those labels are not exactly accurate.
C ONCLUSIONS
In this paper, we investigate how a practitioner interested in learning disentangled representation on some dataset could approach the problem.
We empirically observed that a very limited amount of supervision allows for supervised model selection in the setup of Locatello et al.
We also tested whether with the same amount of supervision, one could obtain better disentanglement if the labels were used for training and observed that it is often the case.
While unsupervised model selection remains a big open challenge for the community, we argue that disentanglement methods could be reliably used in practice as long as a little information about the factors of variation is available.
With this paper, we hope to renew the interest in semi-supervised disentanglement learning which is nowadays less popular than unsupervised training with supervised model selection.
Notably, the impossibility result of Locatello et al.
(2018) does not apply in this setting and our experiments show promising results.
As future work, we aim at testing model selection using the disentanglement metrics with only 10 possibly noisy labelled points over a large number of seeds.
We will also run a large scale comparison, adding supervised losses to other unsupervised methods, in particular β -TCVAE appears to be a promising candidate.
Revisiting semi-supervised approaches that go beyond the simple proof-of-concept of this paper is also a promising direction for future research.
A RCHITECTURE AND SETUP
We build these experiments using the disentanglement lib of Locatello et al.
We add a cross entropy loss to the ﬁrst few dimensions of the mean representation of the encoder.
We test the following values for the hyperparameter in front of the supervised loss: [1 , 2 , 4 , 6 , 8 , 16] .
These values are the same as the unsupervised regularizer.
In all our experiments, we use eitheror 1000 labels.
In the semi supervised models, we used 90% of the labels to train and the rest for validation.
We consider the cases where labels are perfect, noisy ( 10% are random) or binned in ﬁve values.
The disentanglement metrics for testing always use 10 000 labels for numerical stability.
We run the semi-supervised training for 5 different random seeds.
Optimizer, architectures, latent space size, batch size are all identical to the ones used in Locatello et al.
B A DDITIONAL RESULTS
In Figure 3 we show the rank correlation between the different disesntanglement metrics and the ones we use to tune hyperparameters.
We can clearly see a consistent correlation (except with SAP 100) between the metrics across the different datasets.
We conclude that using these supervised metrics to tune hyperparameters of unsupervised models is possible even if only few labels are available.
In table 4, we present how often the simple semi-supervised method beats supervised model selection using 1000 samples.
We can see that the gap between the two approaches is bigger than withsamples as we are able to incorporate more supervision while training.
In table 5, we see a similar trend, this time comparing against all the unsupervised methods and not only β -VAE.
In tables 6we present the median values of the disentanglement metrics achieved after model selection using the validation loss for the semi-supervised model with perfect labels and the MIG.
We assume to have available 100 samples to compute the MIG and 10 for the validation loss as the other 90 are used for training.
In Figures 5-10, we show the conﬁdence bound for these results.
In Figure 4, we can see that the semi-supervised model is relatively robust to noisy and simpliﬁed labels.
[Figure]
Disentanglement
MIG
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure] [Figure]
[Figure]
, 7=VAE).
[Figure] [Figure]
[Figure]
N o
i
[Figure]
i
[Figure]
a t a s e t
[Figure]
c r e a m
[Figure]
dSp r i t
[Figure]