A F RAMEWORK FOR THE Q UANTITATIVE E VALUATION OF D ISENTANGLED R EPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent AI research has emphasized the importance of learning disentangled representations of the explanatory factors behind data.
Despite the growing interest in models which can learn such representations, visual inspection remains the standard evaluation metric.
While various desiderata have been implied in recent deﬁnitions, it is currently unclear what exactly makes one disentangled representation better than another.
In this work we propose a framework for the quantitative evaluation of disentangled representations when the ground-truth latent structure is available.
Three criteria are explicitly deﬁned and quantiﬁed to elucidate the quality of learnt representations and thus compare models on an equal basis.
To illustrate the appropriateness of the framework, we employ it to compare quantitatively the representations learned by a state-of-the-art model (InfoGAN) and those learned by a baseline model (PCA).
I NTRODUCTION
To gain a conceptual understanding of our world, models must ﬁrst learn to understand the factorial structure of low-level sensory input without supervision (Bengio et al., 2013; Lake et al., 2016;
Higgins et al., 2017).
As argued in several notable works (Desjardins et al., 2012; Bengio et al., 2013; Chen et al., 2016; Higgins et al., 2017), this understanding can only be gained if the model learns to disentangle the underlying explanatory factors hidden in unlabelled input.
A disentangled representation is generally described as one which separates the factors of variation, explicitly representing the important attributes of the data (Desjardins et al., 2012; Bengio et al., 2013; Cohen & Welling, 2014b;
Kulkarni et al., 2015; Chen et al., 2016; Higgins et al., 2017).
For example, given an image dataset of human faces, a disentangled representation may consist of separate dimensions (or features) for the face size, hairstyle, eye colour, facial expression, etc.
Ultimately, we would like to learn representations that are invariant to irrelevant changes in the data.
However, the relevant downstream tasks are generally unknown at training time and hence it is difﬁcult to deduce a priori which features will be useful.
Thus, the most robust method is to disentangle as many factors of variation as possible, discarding as little information as possible (Desjardins et al., 2012; Bengio et al., 2013).
Despite the expanding literature on models which seek to learn disentangled representations (Desjardins et al., 2012;
Reed et al., 2014; Zhu et al., 2014; Cheung et al., 2014; Larsen et al., 2015;
Makhzani et al., 2015; Yang et al., 2015; Kulkarni et al., 2015; Whitney et al., 2016; Chen et al., 2016; Higgins et al., 2017; Denton & Birodkar, 2017), visual inspection remains the standard evaluation metric.
While the work of Higgins et al.
(2017) partially addresses this issue (as discussed in section 3) and various deﬁnitions have implied additional desiderata like interpretability (Bengio et al., 2013; Kulkarni et al., 2015; Chen et al., 2016), invariance (Goodfellow et al., 2009; Cohen & Welling, 2014a;b; Lenc & Vedaldi, 2015) and equivariance (Kivinen & Williams, 2011; Lenc &
Vedaldi, 2015; Jayaraman & Grauman, 2015), current research generally lacks a clear metric for quantitatively evaluating and comparing disentangled representations.
In this work we propose a framework for the quantitative evaluation of disentangled representations when the ground-truth latent structure is available.
To elucidate the quality of learnt representations and thus compare models on an equal basis, desiderata of disentangled representations are explicitly deﬁned and quantiﬁed .
These uniﬁed desiderata help deﬁne the disentangled representations which we seek and remove the need for a subjective visual evaluation by a human arbiter.
To illustrate the appropriateness of this framework, we employ it to compare quantitatively the representations learned by information maximizing generative adversarial networks (InfoGAN, Chen et al. 2016) and those learned by principal components analysis (PCA).
In the remainder of this paper, we begin by detailing the theoretical framework and how it facilitates the quantitative evaluation of disentangled representations.
Next we review related desiderata and metrics for evaluating disentangled representations.
Finally, we describe the dataset and model speciﬁcs before presenting the experimental results.
T HEORETICAL F RAMEWORK
Models for disentangled factor learning seek a compact data representation or code c of dimension
D , which consists of disentangled and interpretable latent variables.
For synthetic data, the Kdimensional generative factors z are designed to be an ideal such representation.
Thus if D = K the ideal disentangled code c ∗ should be some (scaled) permutation of z , i.e. they should be related by a generalized permutation matrix (or monomial matrix 1 ).
If D > K , one would expect to obtain this monomial structure along with a number of ‘dead’ or irrelevant units in c which are not predictive of / informative about z .
Thus, we can quantitatively evaluate the codes learned by a given model M using the following steps:
Train M on a synthetic dataset with generative factors
z 2.
Retrieve c for each sample x in the dataset ( c = M ( x ) ) 3.
Train regressor f to predict z given c ( ˆ z = f ( c ) ) 4.
Quantify f ’s deviation from the ideal mapping and the prediction error
We now detail the proposed evaluation metrics, i.e., steps 3 and 4.
We train K regressors to predict the value of K generative factors.
The regressor f j predicts z j given c , that is, it learns a mapping f j ( c ) :
R D → R 1 .
We begin with linear regressors and encourage a sparse mapping between c and z with an (cid:96) 1 regularisation penalty (lasso regressors).
With the inputs and targets normalised to have zero mean and unit variance, the magnitude of the resulting regression weights rank the learnt code variables c 0 , . . .
, c D − 1 in order of relative importance to the prediction.
That is, they reveal which code variables capture information about a given generative factor.
Thus, we deﬁne the matrix of relative importances R as R ij =
| W ij | for linear regression, where R ij denotes the relative importance of c i in predicting z j and | W ij | denotes the magnitude of the weight used to scale c i in predicting z j .
This allows us to explicitly deﬁne and quantify three criteria of disentangled representations or codes which are implicit in recent deﬁnitions (Desjardins et al., 2012; Bengio et al., 2013;
Kulkarni et al., 2015; Chen et al., 2016; Higgins et al., 2017), namely disentanglement , completeness and informativeness .
Disentanglement.
The degree to which a representation factorises or disentangles the underlying factors of variation, with each variable (or dimension) capturing at most one generative factor.
The disentanglement score D i of code variable c
i is quantiﬁed by D i =
(1 − H K ( P i. )) , where
H K ( P i. ) =
− (cid:80) K − 1 k =
P ik log K P ik denotes the entropy and P ij =
R ij / (cid:80) K − 1 k =
R ik denotes the ‘probability’ of c i being important for predicting z j .
If c i is important for predicting a single generative factor, the score will be 1.
If c i is equally important for predicting all generative factors, the score will be 0.
D i can be visualised by examining row i of the Hinton diagrams as in Figure 3.
In order to account for dead or irrelevant units in c , relative code variable importance ρ
i
=
(cid:80) j R ij / (cid:80) ij R ij is used to construct a weighted average (cid:80) i ρ
i D i expressing overall disentanglement.
If a code variable c
i is irrelevant for predicting z , then its ρ i (and thus contribution to the overall disentanglement) will be near zero.
Completeness.
The degree to which each underlying factor is captured by a single code variable.
The completeness score C j in capturing generative factor z j is quantiﬁed by C j = (1 −
H D
( ˜ P .j )) , where H D ( ˜ P .j )
= − (cid:80) D − 1 d =
˜
P dj log D ˜ P
ij denotes the entropy of the ˜ P .j distribution.
If a single code variable contributes to z j ’s prediction, the score will be 1 (complete).
If all code variables equally contribute to z j ’s prediction, the score will be 0 (maximally overcomplete).
C j can be visualised by examining column j of the Hinton diagrams as in Figure 3.
Informativeness.
The amount of information that a representation captures about the underlying factors of variation.
To be useful for natural tasks which require knowledge of the important attributes of the data (e.g. object recognition), representations must ultimately capture information about the underlying factors of variation (Bengio et al., 2013; Chen et al., 2016).
The informativeness of code c about generative factor z j is quantiﬁed by the prediction error E ( z j , ˆ z j ) (averaged over the dataset), where E is an appropriate error function and ˆ z j = f j ( c ) .
It is important to note that the prediction error E ( z j , ˆ z j ) , and thus this informativeness metric, is dependent on the capacity of f , with linear regressors only capable of extracting information about z in c that is explicitly represented (linearly encoded).
Hence this informativeness metric is also dependent on a model’s ability to explicitly represent information about z in c , which in turn is dependent on the model’s ability to disentangle the underlying factors of variation ( z ).
Thus the informativeness metric has some overlap with the disentanglement metric, with the size of the overlap determined by the capacity of f (no overlap with inﬁnite capacity).
While the disentanglement score quantiﬁes the number of generative factors captured by a given code variable, the completeness score quantiﬁes the number of code variables which capture a given generative factor.
Together, these scores quantify the deviation from the ideal one-to-one mapping between z and K of the dimensions in c .
Figure 1 illustrates this idea.
Despite the overlap between the disentanglement and informativeness metrics with low-capacity linear regressors, these are ultimately distinct criteria.
While disentanglement requires each code variable in c to be only perturbed by changes in a single z , informativeness requires these perturbations to be systematic and thus informative .
This motivates the use of non-linear regressors.
We use random forest regressors due to their inbuilt ability to determine the relative importance of each feature to a given prediction, thus allowing us to quantify the degree of disentanglement and completeness as before by directly specifying the matrix of relative importances R .
Random forests average the predictions and feature importances from each decision tree in the ensemble.
The number of times a tree chooses to split on a particular input variable determines its importance to the prediction.
Thus, the relative importance of each input variable c
i is given by the number of cases split on c i over the total number of splits (Breiman et al., 1984).
While the ideal code would be able to explicitly represent each generative factor with a single variable, models with generic priors cannot be expected to learn such complete and explicit codes.
For example, generative factors which are drawn from a distribution on a circle cannot be accurately captured by single code variables on which unwrapped prior distributions are imposed.
Thus, with generic priors like the standard normal, information about such generative factors may be nonlinearly encoded across multiple code variables.
Empirical results in Appendix B, Appendix C and (Higgins et al., 2017, ﬁg. 7) support this idea, with several code variables resembling non-linear functions (like the sine and cosine) of the object azimuth.
This further motivates the use of nonlinear regressors.
Our criteria assume that it is possible to recover the latent factors z from the data.
If the data x depends on a linear combination of (some of) the underlying z ’s with a spherically symmetric distribution, then it will only be possible to recover these components up to a rotation matrix.
This is the well-known issue of the rotation of factors in the linear factor analysis model (see e.g., Mardia,
Kent, and Bibby 1979, sec. 9.6), and also leads to the condition in independent components analysis (ICA) that at most one of the z ’s can be Gaussian (Hyv¨arinen et al., 2001).
In this case, the infor- mativeness metric remains valid but the disentanglement and completeness metrics do not as they are dependent on the arbitrary rotation which determines c ’s alignment with z .
Although z may be used to compute the rotation matrix which best aligns c and z , we ultimately wish to evaluate models which will not have access to z at test time.
R ELATED W ORK
The question of how well a learned representation c matches the true generative factors z has been considered in the ‘square’ case of independent components analysis (ICA), where D = K .
In the
ICA case, the data is generated as x =
A z and the learned representation is obtained as c = W x , where A is the mixing matrix and W is the learned ‘un-mixing’ matrix.
Ideally P = WA will be equal to a permutation matrix.
Yang & Amari (1997, sec. 6.1) propose an error metric to assess how close P is to a permutation matrix 2 .
This metric takes the form
[Equation]
summing two terms which have similar goals to our disentanglement and completeness scores respectively, although expressed by comparing with the maximum value in the row or column, rather than via an entropic measure.
Note that, due to the linear structure of ICA, there is no explicit mapping between c and z .
We report separate scores as they capture distinct criteria and go beyond this metric by handling the non-square case when D > K .
Predicting z from c has also been considered previously.
Higgins et al.
(2017, sec.
A.5) use a linear classiﬁer to predict discrete settings of z and thus quantify the amount of explicit information about z in c , albeit with an unnecessary discretisation.
Higgins et al.
(2017, sec. 3) also propose a disentanglement metric.
With this method, one of the generative factors say z k is held ﬁxed, and pairs of x ’s are drawn, generated with different random z ’s except for the ﬁxed z k .
Pairwise absolute differences of the resulting codes | c 1 − c 2 | are then computed and averaged over repetitions before being used to train a linear classiﬁer to predict which generative factor was held ﬁxed.
In our view this is unnecessarily cumbersome—by setting up a regression problem to predict z from c as we have done, the structure of the R ij matrix can be interrogated to quantify the degree of disentanglement.
In addition, this facilitates the quantiﬁcation of additional criteria, namely completeness and informativeness, without needing to generate any additional datasets.
Glorot et al.
(2011, ﬁg. 3) predict z from c using a lasso regressor but only to qualitatively assess disentanglement, visually assessing the overlap of important features for the separate tasks of domain recognition and sentiment classiﬁcation.
Karaletsos et al.
(2015) do so with an unspeciﬁed regressor, thus quantifying informativeness.
In addition, they devise a quantitative metric to determine a model’s ability to disentangle the underlying factors of variation in images.
In particular, they predict the order of query-speciﬁc oracle triplets of images, where the order indicates image similarity with respect to a query (e.g., ‘Where is the light condition most similar in terms of azimuth?’).
However, the proposed metric is speciﬁcally designed to evaluate their ‘oracle-prioritized belief network’ and thus overly cumbersome to be used as a generic disentanglement metric.
Properties such as invariance and equivariance have been proposed as desiderata for representations or codes (Goodfellow et al., 2009; Kivinen & Williams, 2011; Cohen & Welling, 2014b; Lenc &
Vedaldi, 2015; Jayaraman & Grauman, 2015).
In our view these qualities arise naturally from a properly disentangled and informative code.
Consider, for example, the code of an object which consists of separate variables for its class (e.g., cup, bottle, banana etc.)
, position, pose, texture etc.
If the object is translated, its position code variable(s) will transform accordingly (equivariance), but other code variables will remain invariant.
E XPERIMENTS
We employ the framework to compare quantitatively the codes learned by InfoGAN (Chen et al., 2016) to those learned by PCA.
The scores for each criteria quantify how superior InfoGAN’s codes are to PCA’s and also explain why this is the case—illustrating the appropriateness of the framework.
[Figure]
4. D ATA
The graphics renderer described in (Moreno et al., 2016) was employed to generate 200 , 000 images of an object (teapot) with varying pose and colour (see Figure 2a).
For simplicity, the camera is centred on the object, the scene background is removed and additional generative factors (shape and lighting) are held constant.
Each generative factor is independently sampled from its respective uniform distribution: azimuth( z 0 ) ∼ U [0 , 2 π ] , elevation( z 1 ) ∼ U [0 , π/ 2] , red( z 2 ) ∼ U [0 , 1] , green( z 3 ) ∼ U [0 , 1] , blue( z 4 ) ∼ U [0 , 1] .
In line with recent architectures, we set the image dimensions to be 64 × 64 × 3 .
Disentangled representations should enable a model to perform zero-shot inference, that is, generalise its knowledge beyond the training distribution by recombining previously-learnt factors (Bengio et al., 2013; Higgins et al., 2017).
Thus, we can further evaluate the disentangled representations learned by a given model by quantifying its ability to perform zero-shot inference.
We use the ground-truth values of the generative factors to create two different data distributions.
More speciﬁcally, we isolate all images whose generative factor values lie in a particular range to create a ‘gap’ in the original dataset.
This gap then serves as our zero-shot data containing unseen factor combinations.
Informally, the images in this gap can be described as ‘red’ teapots from ‘above’.
Formally, the generative factors of these images satisfy the following condition:
z 2 > ( z 3 + 0 . 15) and z 2 > ( z 4 + 0 . 15) and z 1 > π 4 .
This dataset contained approximately 20 , 000 images, with (extreme) samples given in Figure 2b.
Note that zero-shot inference is facilitated by disentangled and informative representations, thus is not a core component of our framework, but rather a ‘bonus’.
4. M ODELS
We train both InfoGAN (Chen et al., 2016) and PCA with 6 code variables.
For illustrative purposes, we choose the best of 10 random runs by visual inspection for InfoGAN as we found the model to be quite sensitive to random initialisation.
Note that InfoGAN’s code variables are termed ‘latent codes’ in (Chen et al., 2016) while PCA’s are termed principal components.
Further details on the
InfoGAN model setup are provided in Appendix A.1.
4. R ESULTS
As each target is normalised to have a standard deviation of 1, the root-mean-square error (RMSE) in predicting each target is naturally normalised relative to the constant regressor which guesses the expected value of the targets.
Hence, we report the normalised root-mean-square error (NRMSE) in predicting each target.
Tables 1a and 2a present the disentanglement scores for the lasso and random forest regressors respectively.
With both regressors, the variables in InfoGAN’s code ( c − InfoGAN ) achieve much higher disentanglement scores than those in PCA’s code ( c − PCA ), with each variable in c − InfoGAN closer to capturing a single generative factor.
That is, c − InfoGAN is more disentangled .
Figure 3 helps to visualise the disentanglement and identify the generative factors captured by each code variable.
For example, comparing c 0 − PCA and c 0 − InfoGAN in ﬁgure 3 (the ﬁrst rows), it is clear that c 0 − PCA captures information about each generative factor while c 0 − InfoGAN (almost) solely captures information about z 4 .
The lack of disentanglement within c − PCA indicates that PCA is unable to separate the factors of variation ( z ) in the data, ultimately preventing it from generalising to data with unseen factor combinations.
This is illustrated by the high zero-shot
NRMSE of c − PCA in tables 1d and 2d, with c − PCA even outperformed by the constant regressor in predicting z 1 .
In contrast, c − InfoGAN predicts the value of unseen factor combinations reasonably well.
Tables 1c and 2c present the completeness scores for the lasso and random forest regressors respectively.
The high completeness scores of c − InfoGAN reveal that it captures each generative factor with approximately one code variable.
That is, they show that c − InfoGAN is almost complete.
In contrast, the low scores of c − PCA reveal that is it severely overcomplete, using several code variables to capture each generative factor.
Again, Figure 3 helps to identify the generative factors captured by a given code variable and visualise the completeness.
With an ideal representation, Figure 3 would show a single large weight in K rows and each column, indicating a one-to-one mapping between z and K of the dimensions in c .
Figure 3 also helps to explain the low completeness score (overcompleteness) of c − InfoGAN in predicting z 0 , with z 0 clearly captured by a combination of c 1 − InfoGAN and c 3 − InfoGAN .
However, this is still less overcomplete than c − PCA , with each of its constituent variables capturing distinct information about z 0 (see Figure 3b).
Tables 1b and 2b present the test set NRMSEs in predicting z for factor combinations similar to those on which the models were trained.
c − InfoGAN clearly outperforms c − PCA in predicting each generative factor with both regressors.
That is, it is far more informative .
It is worth noting the signiﬁcantly higher error in predicting the azimuth ( z 0 ) from c − InfoGAN .
This can be attributed to: (i) the low-capacity linear regressor being unable to extract the information encoded in c − InfoGAN about the azimuth (indicated by much lower azimuth prediction error of the non-linear regressor in
Table 2b);
(ii) InfoGAN struggles to capture enough information about the azimuth in c − InfoGAN (indicated by the relatively high error in predicting the azimuth with both regressors).
Comparing the results of the linear regressor in Table 1b with those of the non-linear regressor in Table 2b, we see that both codes better predict the generative factors with increased capacity—especially c − PCA .
As discussed in section 2, the prediction error with the non-linear random forest regressor is likely a better quantiﬁcation of informativeness as it is less dependent on the ability of the model to explicitly represent information about z in c .
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Figure] [Figure]
c
[Figure] [Figure]
thus the disentanglement.
Column j illustrates the relative importance of each code variable for predicting z j and thus the completeness.
C ONCLUSION
In this work we have presented a framework for the quantitative evaluation of disentangled representations when the ground-truth latent structure is available.
The quality of learnt representations is elucidated through the explicit deﬁnition and quantiﬁcation of three criteria: disentanglement, completeness and informativeness.
We employed our framework to compare quantitatively the codes learned by InfoGAN to those learned by PCA.
This contrastive comparison demonstrates the appropriateness of our framework, with the three criteria explaining why InfoGAN’s learnt codes are superior to PCA’s and the metric scores quantifying this level of superiority.
While our framework is limited to synthetic datasets where it is possible to recover z , reliable disentanglement is far from solved even in this restricted setting.
Hence, we believe our framework and its constituent metrics take a substantial and important step forward in understanding learned representations.
We will make the code and dataset publicly available on acceptance of this paper, and hope this facilitates further model comparisons and eventually the establishment of quantitative benchmarks for disentangled factor learning.
With many models and datasets, reporting detailed results for both regressors may be quite cluttered.
In such cases, the average criteria scores with the random forest regressor may be an adequate summary to report, with its higher capacity making informativeness and disentanglement more distinct criteria.
While we have focused on image data in this work, future work may explore the applicability of our framework to other types of synthetic data.
A CKNOWLEDGEMENTS
Removed to preserve anonymity.
A E XPERIMENTAL S ETUP A. I NFO GAN
Extending the GAN of Goodfellow et al.
InfoGAN (Chen et al., 2016) splits the generator input into two parts; ‘incompressible noise’ and ‘latent codes’ which target salient semantic features of the data.
The manner in which the generator may use these latent codes is constrained by adding a regularisation term to the GAN objective, representing the mutual information between the latent codes and generated images.
For stability, we use the training objective of the improved
Wasserstein GAN (IWGAN) (Gulrajani et al., 2017) and the 64 × 64 ResNet architecture described in the open source implementation of Gulrajani et al.
We modify this implementation to add InfoGAN’s variational regularisation of mutual information to the IWGAN training objective, splitting the generator input into noise and latent code components before implementing the auxiliary network Q as in (Chen et al., 2016).
The network Q parametrises the approximate posterior over latent codes Q ( c | x ) , with Q ( x ) returning a mean and standard deviation for each continuous (normal) latent code in the factorised posterior Q ( c | x ) .
Thus, to retrieve the (most likely) representation or code c for a given image x , we simply take the means returned by Q ( x ) .
Q shares all convolutional layers with the discriminator or ‘critic’ D , each adding their own ﬁnal output layer.
All hyperparameters of the IWGAN implementation remain unchanged while we found setting the mutual information coefﬁcient λ
= 8 to be sufﬁcient, ensuring that the mutual information penalty was on the same scale as the unbounded WGAN objectives.
For all experiments, we use 6 continuous latent codes and 128 noise variables resulting in a generator input with dimension 134.
Further details on the architecture, hyperparameters and combined training objective are provided in our open-source implementation, to be made publicly available on acceptance of this paper.
A. R EGRESSORS
As performance generally improves with the number of trees n in the ensemble, we ﬁx n = 10 for the random forest regressor.
All other parameters and hyperparameters are ﬁt to a validation set
[Figure]
B G ENERATIVE FACTORS VS . LATENT CODES WITH GENERIC PRIORS
[Figure] Figure 4 plots each generative factor against the corresponding ‘most important’ InfoGAN code variable, as indicated by the lasso regression weight magnitudes and random forest feature importances.
Information about each unwrapped generative factor ( z 1 , z 2 , z 3 , z 4 ) is linearly-encoded in single code variables ( c 5 , c 4 , c 2 , c 0 respectively).
In contrast, distinct information about the azimuth ( z 0 ) is non-linearly encoded across c 1 and c 3 , reinforcing the argument that models with generic priors cannot be expected to learn the most complete and explicit representation of topologically distinct factors of variation.
Furthermore, when InfoGAN was retrained instead with 10 code variables, it used three of these to capture the azimuth (see Figure 5a), each resembling (scaled) sine and cosine functions (see Figure 5b).
C D = 10 , K =
As shown in Figure 5a, several ‘dead’ or redundant code variables ( c 4 , c 7 , c 9 ) enable a high degree of completeness in c − InfoGAN .
Although c − InfoGAN captures z 0 (azimuth) with 3 code variables, c − PCA uses a least 7.
Figure 5b shows that the 3 InfoGAN code variables which capture azimuth resemble scaled versions of sine and cosine functions.
While the regression results for InfoGAN with 10 latent codes were inferior, Figure 5a bodes well for InfoGAN’s ability to learn disentangled and interpretable codes without any prior knowledge about the number of underlying factors of variation (i.e. when trained with excessive latent codes).
We note that further hyperparameter searches and random runs may have yielded better results.
4 z          c
[Figure]
4 z          c
[Figure]
D V ISUALLY ASSESSING DISENTANGLEMENT
[Figure]