I MPROVING THE U NSUPERVISED D ISENTANGLED R EPRESENTATION L EARNING WITH VAE E NSEMBLE
Anonymous authors
Paper under double-blind review
A BSTRACT
Variational Autoencoder (VAE) based frameworks have achieved the state-of-the1 art performance on the unsupervised disentangled representation learning.
A re2 cent theoretical analysis shows that such success is mainly due to the VAE im3 plementation choices that encourage a PCA-like behavior locally on data sam4 ples.
Despite this implied model identiﬁability, the VAE based disentanglement 5 frameworks still face the trade-off between the local orthogonality and data re6 construction.
As a result, models with the same architecture and hyperparameter 7 setting can sometimes learn entangled representations.
To address this challenge, 8 we propose a simple yet effective VAE ensemble framework consisting of multi9 ple VAEs.
It is based on the assumption that entangled representations are unique 10 in their own ways, and the disentangled representations are “alike” (similar up to a 11 signed permutation transformation).
In the proposed VAE ensemble, each model 12 not only maintains its original objective, but also encodes to and decodes from 13 other models through pair-wise linear transformations between the latent repre14 sentations.
We show both theoretically and experimentally, the VAE ensemble 15 objective encourages the linear transformations connecting the VAEs to be triv16 ial transformations, aligning the latent representations of different models to be 17 “alike”.
We compare our approach with the state-of-the-art unsupervised disen18 tangled representation learning approaches and show the improved performance.
I NTRODUCTION
Disentangled representation learning aims to capture the semantically meaningful compositional 21 representation of data (Higgins et al., 2018; Mathieu et al., 2018), and is shown to improve the 22 efﬁciency and generalization of supervised learning (Locatello et al., 2019), reinforcement learning 23 (Watters et al., 2019), and reasoning tasks (van Steenkiste et al., 2019).
The current state-of-the24 art unsupervised disentangled representation learning deploy the Variational Autoencoder (VAE) 25 (Kingma & Welling, 2013; Rezende et al., 2014).
The main challenge is to reduce the trade-off 26 between learning a disentangled representation and reconstructing input data.
Most of the recent 27 works extend the original VAE objective with carefully designed augmented objective to address 28 this trade-off (Higgins et al., 2017; Burgess et al., 2017; Kim & Mnih, 2018;
Chen et al., 2018; 29 Kumar et al., 2017).
A recent study in (Locatello et al., 2018) compared these methods and showed 30 that their performance is sensitive to initialization and hyperparameter setting of the augmented 31 objective function.
Recently, Duan et al.
(Duan et al., 2019) developed an unsupervised model selection method named 33 Unsupervised Disentanglement Ranking (UDR) to address the challenge of hyperparameter search 34 and model selection.
UDR leverages the ﬁnding in (Rolinek et al., 2019) that the implementation 35 choices of VAE encourage a local PCA-like behavior locally on data samples.
As a result, disen36 tangled representations by VAEs are “alike” as they are similar up to signed permutation transfor37 mations.
On the contrary, the entangled representations by VAEs are “unique” as they are similar 38 at least up to non-degenerate rotation matrices.
UDR uses multiple models trained with different 39 initializations and hyperparameter settings, and builds a similarity matrix measuring the pair-wise 40 similarity between the latent variables from different models.
A higher score is given to the model 41 that can match its representations to many others models.
The results show close match between 42 UDR and commonly used supervised metrics, as well as the performance of downstream tasks using 43 the latent representations.
Inspired by the ﬁndings from these studies, we propose a simple yet effective VAE ensemble framework to improve the disentangled representation by VAE.
The proposed VAE ensemble consists ofmultiple VAEs.
The latent variables in every pair of these models are connected through linear layers to force the latent representations in the ensemble to be similar up to a linear transformation.
We show that the VAE ensemble objective encourages these pair-wise linear transformations to converge to trivial transformations, making latent representations of different VAEs in the ensemble tobe “alike”, thus disentangled.
In this paper, we make the following contributions: (1) We introducea simple yet effective VAE ensemble framework to improve the disentangled representation learningusing the original VAE.
(2) We show in theoretical analysis that the linear transformations connecting the latent representations of the individual models in the ensemble tend to converge to trivialtransformations thus encourage disentangled representation, and verify this result with experiments.(3)
We evaluate our approach using the original VAE model, and show the improved state-of-the-artperformance across different datasets.
R ELATED W ORK
Variatioanl Autoencoder is a deep directed probabilistic graphical model consisting of an encoderand a decoder (Kingma & Welling, 2013; Rezende et al., 2014).
The encoder q φ ( z | x ) maps theinput data x ∈ R n to a probabilistic distribution as the latent representation z ∈ R d , and the decoderq θ ( x | z ) maps the latent representation to the data space noted as q θ ( x | z ) , where φ and θ representmodel parameters.
The VAE objective is to maximize the marginalized log-likelihood of data.
Directoptimization of this objective is not tractable and it is approximated by the evidence lower bound(ELBO) as:
[Equation]
In practice, the ﬁrst term is estimated by reconstruction error.
The second term is the KullbackLeibler divergence between the posterior q φ ( z | x ) and the prior p ( z ) commonly chosen as anisotropic unit Gaussian p ( z ) ∼ N (0 , I ) .
Disentangled representation by VAE has achieved the state-of-the-art performance (Higgins et al.
,2017; Burgess et al., 2017; Kim & Mnih, 2018;
Chen et al., 2018; Kumar et al., 2017), despite thefact that the VAE objective only models the marginal distribution of the data instead of the desiredjoint distribution over data and latent variables.
The reason for this success is the implementationchoices of the VAE framework (Rolinek et al., 2019).
In practice, the latent variables in VAE oftenwork in “polarized” modes.
The “passive” mode is deﬁned by µ 2 j ( x ) (cid:28) 1 and σ 2 j ( x ) ≈ 1 , while the“active” mode is deﬁned by σ 2 j ( x ) (cid:28) 1 .
The “passive” latent variables closely approximate the priorand have little effect on the decoder.
The “active” latent variables, on the other hand, are closelyrelated to both the per sample KL loss and the decoder output.
The “polarized regime” enables areformulated VAE objective showing that VAEs optimize a trade-off between data reconstructionand orthogonality of the linear approximation of decoder Jacobian locally around a data sample.
This PCA-like behavior near data points encourages an identiﬁable disentangled latent space by
VAE.
Furthermore, it was suggested that ﬁnding an appropriate “polarized regime” is dependenton the initialization and the hyperparameter tuning of the state-of-the-art approaches.
In this study,we show that the proposed VAE ensemble aligns the “polarized regime” of individual VAE modelstowards the disentangled representation.
Model selection In practice, we often observe neural networks achieve similar performance withdifferent internal representations when trained with the same hyperparameters (Raghu et al., 2017;
Wang et al., 2018; Morcos et al., 2018).
For the unsupervised disentanlged representation, as discussed in (Locatello et al., 2018; Duan et al., 2019), we often observe high variance in the performance from the model trained with the same architecture and hyperparameter setting.
This posesa challenge for choosing the model in practice.
Duan et al.
(2019) proposed Unsupervised Disentanglement Ranking (UDR) to address this challenge.
The extensive empirical evaluations on UDRusing both the supervised metric measurement and the performance of downstream tasks validatesits effectiveness.
They also conﬁrm that disentangled representations are “alike” and entangled representations are unique in their own ways.
The proposed VAE ensemble leverages this ﬁnding.
Identiﬁable VAE Built on the recent breakthroughs in nonlinear Independent Component Analysis (ICA) literature (Hyvarinen & Morioka, 2016; 2017; Hyvarinen et al., 2019), Khemakhem etal.
show that the identiﬁcation of the true joint distribution over observed and latent variables is possible up to very simple transformations (Khemakhem et al., 2019).
They proposed identiﬁable 98 VAE (iVAE) that requires a factorized prior distribution over the latent variables conditioned on an 99 additionally observed variable, such as a class label or almost any other observation.
We believe theproposed VAE ensemble is related to such framework where the latent representation from one VAEmodel can be regarded as the auxiliary observations for another.
Ensemble learning The idea of ensemble learning is to combine multiple learning models (potentially weak learners) to improve the task performance or robustness over a single model (Schapire,1990).
It achieves the improved performance by averaging the bias, reducing the variance thus preventing the over-ﬁtting (Drucker et al., 1994; Breiman, 1996).
Early works in neural networks haveused the ensemble learning to achieve top performance in the related competition (Krizhevsky et al.
,2012; Simonyan & Zisserman, 2014).
In this work, we apply the ensemble learning to enforce thealignment among the latent representations of different models.
This results in latent representationsthat are similar among each other in the ensemble up to a trivial transformation.
T HE VAE E NSEMBLE F RAMEWORK
As illustrated in Figure 1, the proposed VAE ensemble consists of n original VAE models with thesame architecture but different initializations.
It also consists of n × ( n − 1) linear layers connectingthe latent representations of every two VAE models.
Each model in the ensemble maintains itsoriginal VAE objective as Eq.
In addition, l 2 loss is used to force mapping between latentrepresentations via pair-wise linear layers (cross-model linear transformation).
The decoder of each
VAE model generates the input reconstruction from not only their corresponding encoder (withinmodel reconstruction), but also the linearly transformed encodings from other encoders (cross-modelreconstruction).
Overall, the VAE ensemble is trained with the following objective:
[Equation]
where n is the number of models in the ensemble; φ := ( φ ij ) is the encoders parameters where φ ijrepresents the encoder of VAE
i and its associated linear layer mapping the latent representation from
VAE i to VAE j
(notice that φ ii represents the encoder parameters of VAE i only and its associatedlinear transformation can be be regarded as an identity transformation); θ :=
( θ i ) represents the decoder parameters of VAE i ; and z jj represents the latent representation of VAE j while z ij representsthe linearly transformed latent representation from VAE i to VAE j .
γ is a hyperparameter to balancethe effect of the estimation error between the latent representaitons.
p ( z ii ) ∼ N (0 , I ) is assume tobe the prior as deﬁned in the original VAE objective.
Comparing to the original VAE objective in Eq. 1, the objective of each individual VAE model inthe ensemble, L (cid:48)
V AE , can be written as:
[Equation]
where n is the number of models in the ensemble, φ j stands for the parameters of the encoder andits linear transformation layers from other VAEs, and z j represents the linear transformed latentrepresentation of the encoding from other encoders.
In this form, the VAE ensemble regularizes each VAE model with additional terms on the encoderas γ E q φj ( z j | x ) (cid:107)
z
j − z (cid:107) 2 , and on the decoder as (cid:80) nj =1 E q φj ( z j | x ) [log q θ ( x | z j )] .
These regularizations directly constrain the latent representations among different VAE models in the ensemble tobe similar.
In particular, for a given input data, (cid:107) z
j − z (cid:107) 2 encourages the encoders to generatesimilar encodings up to the linear transformations; and (cid:80) nj =1 E q φj ( z j | x ) [log q θ ( x | z j )] emphasizesthe similar effect on the data reconstruction from the latent variables such that the decoders canreconstruct the input data with both the original encoding z and the linearly transformed encodingz j .
As we shall discuss in the next section, together these regularizations encourage similar latentrepresentation up a trivial transformation by different models in the ensemble.
We also introduce the hyperparameter γ to balance the trade-off between these two regularizations:higher value forces closer mapping between the encoders and reduce the cross-model reconstructionerror of the decoders; lower value relaxes the mapping between the encoders and increases the crossmodel reconstruction error of the decoders.
As we show in Section 5, both components are importantand balancing the trade-off between them is important as the ensemble size increases.
Computational complexity It is a common practice to train a number of seeds per hyperparametersetting for the current state-of-the-art unsupervised disentanglement VAE models (Locatello et al. ,2018; Duan et al., 2019). Comparing to training n original VAEs, the proposed VAE ensemblerequires additional n × ( n − 1) linear layers. While this addition does not increase the size of themodel much, the estimation of the linear transformations loss and the cross-model reconstructionlosses grow with n × ( n − 1) , which may be computationally expensive especially when n is large.
That being said, the results in Section 5 show that the VAE ensemble achieves more stable resultscomparing to the current state-of-the-art models.
Also, its computation is highly parallelisable.
In this section, we present the theoretical analysis on why the proposed VAE ensemble can improvethe disentangled representation.
We start with analysing the l 2 objective in Eq.
2 of the pair-wiseliner transformations in the VAE ensemble, and show that: (1) the pair-wise linear transformationsencourage similar “polarized” regime (see Sec. 2) among the VAEs in the ensemble; (2) the lineartransformations are close to the orthogonal transformations.
Based on these two properties, we thendiscuss how the cross-model reconstructions by the VAE ensemble encourage learning a disentangled representation over its entangled counterpart.
4. T HE E FFECT OF L INEAR T RANSFORMATION BETWEEN L ATENT R EPRESENTATIONS
Let VAE i and VAE j be two different VAE models in the ensemble, and M ji be the linear transformation that maps the latent representation of a given input x by VAE j to the one by VAE i , as z j ( x ) ∼ N ( µ j ( x ) , diag ( σ j ( x ) 2 )) to z i ( x ) ∼ N ( µ i ( x ) , diag ( σ i ( x ) 2 )) .
In the following we remove the input notation from the VAE latent representations for simplicity (i.e. z j ( x ) is simpliﬁedas z j ), while keeping in mind that the analysis is based on the local latent representation of a giveninput x .
For VAE j , the l 2 term of the VAE ensemble loss in Eq.
2 aims to ﬁnd M ji and z j that minimize
E (cid:107) z
i − M ji z j (cid:107) 2 , where the expectation is over the stochasticity of VAE j .
We can write z i and z j as z i = µ i + (cid:15) i and z j = µ j + (cid:15) j , where (cid:15) i ∼ N (0 , diag ( σ 2 i )) and (cid:15)
j ∼ N (0 , diag ( σ 2 j )) .
Henceusing bias-variance decomposition, the l 2 term can be written as:
[Equation]
where the constant C 1 arises from the fact E z i (cid:107) µ i − z i (cid:107) 2 does not depend on M ji and
z j . Eq.
4 consists of a deterministic component of (cid:107) µ
i − M ji µ j (cid:107) 2 and a stochastic component of E (cid:15) j (cid:107) M ji (cid:15) j (cid:107) 2 .
The deterministic component can be minimized by adjusting the parameters in VAE j such that itsmean encoding µ j is optimized for any given M ji .
This simpliﬁes our analysis to focus on thestochastic component.
Between M ji and (cid:15) j in this stochastic component, we separately optimizeone while having the other ﬁxed.
We start with ﬁxed M ji and optimizing for (cid:15) j ∼ N (0 , diag ( σ 2 j )) .
Notice that σ 2 j is associated with
VAE j objective.
In (Rolinek et al., 2019), the VAE objective is reformulated into the deterministicreconstruction, the stochastic reconstruction and the KL loss.
The last two components deﬁne thestochastic loss of VAE.
It is formulated as:
[Equation]
where X represents the dataset, D represents the local linear approximation of the Jacobian of thedecoder with singular value decomposition as D = U Σ V T .
Furthermore, the KL loss L KL =12 (cid:80) dk
=1 ( µ 2 jk + σ 2 jk − log σ 2 jk − 1) can be simpliﬁed as L ≈ KL = 12 (cid:80) k ∈ “
[Equation]
” ( µ
[Equation]
jk − log
[Equation]
jk
[Equation]
1)based on the “polarized” regime of VAE.
(Rolinek et al., 2019) shows that σ 2 j act as the precisioncontrol allowed for each latent variable where more inﬂuential ones receive more precision.
Combining the stochastic loss of the linear transformation in Eq.
4 and the stochastic loss of the original
VAE in Eq.
5, the overall stochastic loss on σ 2 j can be formulated as:
[Equation]
where σ 2 jk is the k th element of σ 2 j .
Here we further simplify L KL with the L ≈ KL upto additive constants C 2 when µ j is ﬁxed.
In addition to the precision control of σ 2 j on
VAE j , this objective also aims to ﬁnd an optimal distribution of σ 2 j that aligns the “polarizedregime” among different VAEs.
To see why, let c k be the k th column of M ji , we then have
E (cid:107) M ji (cid:15) j (cid:107)
=
(cid:80) k (cid:107) c
k (cid:107) 2 σ 2 jk .
The Arithmetic-Mean–Geometric-Mean (AM/GM) inequality suggests that (cid:80) k (cid:107) c k (cid:107) 2 σ 2 jk ≥ n (cid:16) (cid:81) k (cid:107) c k (cid:107) 2 σ 2 jk (cid:17) 1 /n = n (cid:16) (cid:81) k
(cid:107) c k (cid:107) 2 (cid:17) 1 /n exp( − C ) , where the equality is achieved when (cid:107)
c m (cid:107) 2 σ 2 jm
=
(cid:107)
c n (cid:107) 2 σ 2 jn for any m (cid:54) = n .
This suggests that latent variableswith high (cid:107) c k (cid:107) 2 mapping from z j to z i will have smaller variance.
Hence, these latent variables in z j 198 are encouraged to stay in the “active” mode.
On contrary, the latent variables that do not share similar generative factors between z j and z i will be assigned larger variance, and being pushed towardsthe “passive” mode.
Now we ﬁx the optimal distribution of σ 2 j , and optimize for M ji .
Since (cid:15) j ∼ N (0 , diag ( σ 2 j )) , thisobjective can be understood as optimally rotating the latent space of VAE j such that the stochasticcomponent in Eq.
4 is minimized.
Speciﬁcally, we have the following objective:
[Equation]
(7) where R is an orthogonal transformation.
Let c (cid:48)
k be the k th column of M ji R .
Similar as before, the
AM/GM inequality suggests (cid:13)(cid:13)
M ji R T (cid:15) j
(cid:13)(cid:13)
= (cid:80) k
(cid:107) c (cid:48)
k (cid:107) 2 σ 2 jk ≥ (cid:81) k
(cid:107) c (cid:48)
k (cid:107) 2 exp( − C 3 ) .
Hadamard’sinequality suggests that (cid:81) k
(cid:107) c (cid:48)
k (cid:107) 2 ≥ | det( M ji R ) | , and the equality is satisﬁed when c (cid:48)
k s are pairwise orthogonal.
This can be understood from the geometric perspective where (cid:81)
k (cid:107) c (cid:48) k (cid:107) 2 gives anupper bound on Volume ( (cid:8) M ji R T x : x ∈ [0 , 1] d (cid:9) ) .
As a result, the optimization of M ji will lead toan orthogonal transformation.
Together the optimization of Eq.
6 and Eq.
7 encourages the alignment of the “polarized regime” among different models under orthogonal linear transformations.
They force different models in the ensemble to capture the same mixture of the generative factors.
In the next section, we discuss the effect of the cross-model reconstruction in the VAE ensemble thatencourages the disentangled representation over the entangled ones.
4. T HE E FFECT OF CROSS - MODEL R ECONSTRUCTION
In an entangled representation, each latent variable captures a mixture of generative factors in itsunique way.
Since different generative factors typically have different effects on data variations(Duan et al., 2019), the orthogonal transformation from one entangled representation z j to anotherone z i introduces different encoding variance.
Some of the transformed latent variables in M ji z jcarry larger variance comparing to the corresponding ones in z i .
This discrepancy leads to largercross-model reconstruction of VAE i than the within model reconstruction.
This error forces both
VAE i and VAE j to adjust their representations until the effect on the data reconstruction by individual latent variables matches between M ji z j and z i .
The process applies to all models in theensemble and eventually leads to a one-to-one mapping of latent variables between different models, where M ji becomes a trivial transformation (signed permutation matrix).
In particular, if one ofthe models in the VAE ensemble learns a disentangled representation, other models in the ensemblewill converge to it.
This is because the orthogonal transformation from an entangled representationto a disentangled representation introduces larger cross-model encoding variance due to the mixtureof different generative factors in the former, thus a larger cross-model reconstruction by the disentangled model.
On contrary, the orthogonal transformation from a disentangled representation to anentangled representation would not introduce larger cross-model encoding variance than the withinmodel encoding, thus similar cross-model reconstruction as within model reconstruction by the entangled model.
Such a gap encourages the entangled representations to align with the disentangledrepresentation.
We illustrate the geometric interpretation of such a case in Appendix C.
From these discussions, we conclude that the VAE ensemble encourages different individual models to capture similar generative factors, thus learn representations that are “alike” up to a trivialtransformation.
In the next section, we verify these analytic results with experiments.
E XPERIMENTS
Our experiments are designed to conﬁrm the discussions in the previous sections.
Particularly weask the following questions: Q1:
Do the linear transformations in the ensemble converge to trivialtransformation?
Q2:
Do the VAEs in the ensemble work in similar “polarized” regime?
Q3:
Does
VAE ensemble improve the unsupervised disentangled representation learning, and what is the effect of ensemble size? Q4: What are the effects of the cross-model reconstruction loss, the lineartransformation loss and the hyperparameter γ in the VAE ensemble objective?
We analyze the inner working of the proposed VAE ensemble using the benchmark dSprite dataset(Matthey et al., 2017) with fully known generative processes, and the real-world CelebA dataset (Liuet al., 2015) with unknown generative process.
Furthermore, for dSprites dataset, we compare ourresults with the original VAE model and the state-of-the-art disentanglement VAE models includingβ -VAE (Higgins et al., 2017), FactorVAE (Kim & Mnih, 2018), TC-VAE (Chen et al., 2018) and
DIP-VAE (Kumar et al., 2017). We use two widely used supervised metrics including FactorVAEmetric (Kim & Mnih, 2018) and DCI Disentanglement scores (Eastwood & Williams, 2018) as the quantitative measurements. They are shown to correlate with other common supervised
metrics(Locatello et al., 2018). For example, FactorVAE metric and β -VAE metric (Higgins et al., 2017)capture similar notions, while DCI Disentanglement and Mutual Information Gap (MIG) (Chenet al., 2018) capture similar notions. In addition, DCI Disentanglement is closely related to theunsupervised model selection method UDR (Duan et al., 2019).
For CelebA dataset, we show thelatent traversal visulization as a qualitative measurement in Appendix E. We provide the details ofthe experiments in Appendix D.
Q1:
We use the Distance to Orthogonality (DtO) (Rolinek et al., 2019) to check if the linear transformations in the ensemble converge to a signed permutation matrix during training.
DtO is the
Frobenius norm of the difference between a matrix M and its closest signed permutation matrix
P ( M ) .
It is solved with mixed-integer linear programming (MILP) formulation.
The details on
DtO can be found in Appendix B.
In Figure 2, we show the DtO estimation of the linear transformations in the VAE ensemble of different ensemble size for the dSprite dataset.
We show the meanand standard deviation of DtO across all linear transformations over 10 different runs.
Furthermore,we compare these results with a baseline measurement where DtO is estimated for the linear transformations between the mean latent representations of well-trained individual models.
Speciﬁcally,we use ten well-trained individual models and report the mean and standard deviation of the DtOestimations.
As seen in the ﬁgure, the VAE ensemble models with different ensemble size all approach to trivial transformations between the individual models, while other VAE models do nothave such property. In Fig.
6, we show that during training, the VAE ensemble remains maintainslow DtO while the original VAEs do not have such property.
A similar result for models trained onthe CelebA dataset with different latent dimensions is shown in Fig. Further discussion on theseresults are provided in Appendix E.
Q2:
To check if the models in the VAE ensemble work in similar “polarized” regime, we estimatethe relative error between L KL and L ≈ KL as ∆ =
L KL − L ≈ KL L KL for each latent variable.
Smaller ∆indicates closer matching between L KL and L ≈ KL of a latent variable, thus more “active”.
Figure3(a) and 3(b) show log(∆) of the 10 latent variables of individual models in VAE E 2 and VAE E278 with different γ settings trained on the dSprite dataset.
The results show that individual modelsin the ensemble do work in similar ”polarized regime’.
In Figure 3(a), we also compare the VAEensemble with the β -VAE where β = 4 .
This setting was found previously to be the optimal settingfor the dSprite data for β -VAE (Higgins et al., 2017).
We see that the VAE ensemble encouragesmore “active” latent variables than β -VAE.
When we compare Fig. 3(a) and 3(b), we see that asthe ensemble size increases, individual models are forced to have more “active” latent variables bydecomposing the generative factors.
This can be observed in the latent traversals shown in Appendix
E. The dSprites dataset contains ﬁve ground truth generative factors.
The VAE E 2 models canhave up to eight “active” latent variables depending on input, and these representations capture adecomposition of the ground truth generative factors.
[Figure]
[Figure]
Individual Model VAE Ensemble (VAE E)
VAE 0.635 ± 0.
γ = γ = γ = β -VAE ( β =4) 0.665 ± 0. VAE E0.711 ± 0.
0.736 ± 0. 0.741 ± 0. FactorVAE ( γ =40) 0.764 ± 0. VAE E0.794 ± 0.
0.792 ± 0 .0.821 ± 0.
DIP-VAE-I ( λ od =5) 0.638 ± 0.
VAE E0.833 ± 0.
0.790 ± 0 .0.800 ± 0.
DIP-VAE-II ( λ od =5) 0.676 ± 0.
VAE E0.828 ± 0.
0.786 ± 0 .0.739 ± 0.
TC-VAE ( β =4) 0.808 ± 0.
DCI-Disentanglement Metric Individual Model VAE Ensemble (VAE E) VAE 0.143 ± 0.
γ = γ = γ = β -VAE ( β =4) 0.198 ± 0. VAE E0.176 ± 0. 0.243 ± 0. 0.201 ±
[Table] Q3: In Table 1, we compare the disentangled representation performance between the proposed VAE ensemble and the state-of-the-art models.
For the VAE ensemble, we report the performanceof the ﬁrst model in the ensemble.
We also report the results for the VAE ensemble with differentensemble size and γ values.
As shown in the table, the VAE ensemble signiﬁcantly improves theperformance over the original VAE model.
In many settings, the VAE ensemble achieves similaror better performance over the state-of-the-art models.
In Table 2, we evaluate the consistencyamong the models in the ensemble by reporting the standard deviation of the evaluation metricsusing different models in the same ensemble.
The small values conﬁrm that different models inthe ensemble learn similar latent representations.
Furthermore, Table 1 shows the joint effect ofensemble size and γ setting.
When γ = 1 , the performance of VAE ensemble increases as theensemble size increases, indicated by the higher mean and smaller variance of both the FactorVAEand DCI Disentanglement metrics.
This behavior is consistent with the characteristic of ensemblelearning where the increase in performance becomes smaller as the size of ensemble increases.
However, as γ increases, having larger ensemble size can reduce the performance.
We believe this [Table] is due to the increased difﬁculty of balancing between the cross-model and within model objectivesof VAE ensemble for larger ensembles.
The reduced alignment of the latent representations amongdifferent models can also be seen in Table 2 where difference in the performance among individual in the increases as ensemble size Q4: We conduct the ablation study to further understand the effect of the linear transformation lossand the cross-model reconstruction loss in the VAE ensemble objective. As shown in Fig. 4, removing either component leads to a lower FactorVAE metric for the VAE ensemble.
Without the lineartransformation loss, the performance of VAE ensemble decreases signiﬁcantly across different ensemble sizes. Without the cross-model reconstruction loss, the performance of VAE ensemble alsodecreases but the gap becomes smaller as γ increases.
This matches the discussion in Section 3 thathigher γ forces closer mapping between the encoders and reduce the cross-model reconstruction error of the decoders. However, this also reduces the effect of cross-model reconstruction as discussedin Section 4.2. A similar result is also found for the DCI Disentanglement metric as shown in Fig.8 in Appendix E. Overall, the results from the ablation study conﬁrms the importance of both thelinear transformation loss and the cross-model reconstruction loss in the VAE ensemble objective.  C ONCLUSION
In this study, we propose a simple yet effective VAE ensemble framework consisting of multipleoriginal VAEs to learn disentangled representation.
The individual models in the ensemble areconnected through linear layers that regularize both encoders and decoders to align the latent representations to be similar up to a linear transformation.
We show in theory that the regularization bythe VAE ensemble forces the linear transformations to be trivial transformations and show improvedperformance on the unsupervised disentangled representation learning.
The theoretical discussionin Section 4 is based on the original VAE objective, and our experiments also focus on the ensemblewith original VAE.
We believe such framework can be extended to other disentangled VAE models,or even a mixture of different VAE models, as long as the regularization by the ensemble does notconﬂict with the augmented objective of these models.
A T ECHNICAL L EMMAS
In this section, we give the lemmas used in the theoretical discussion in Section 4.
Lemma 1.
(Jensen’s inequality)
If g ( x ) is a convex transformation on x , then this convex transformation of a mean g [ E ( x )] is less than or equal to the mean of the convex transformed
value E [ g ( x )] ;it is a simple corollary that the opposite is true of concave transformations.
Lemma 2.
(AM-GM inequality)
As an extension of Jensen’s inequality, given a list of non-negativereal numbers x 1 , x 2 , . . .
, x n , the arithmetic mean of this list 1 n (cid:80) ni
=1 x i is greater than or equalto the geometric mean of the same list ( (cid:81) ni
=1 x i ) 1 n ; and further, the equality holds if and only ifx 1 = x 2 = · · · = x n .
Lemma 3.
(Hadamard’s inequality).
if M is the matrix having columns c i , then | det( M ) | ≤ 411 (cid:81) ni =1 (cid:107) c i (cid:107) ; and the equality in Hadamard’s inequality is achieved if and only if the vectors areorthogonal.
B D ISTANCE TO O RTHOGONALITY (D T O)
In this section, we introduce the detail of Distance to Orthogonality (DtO) that is used in our experiment to check if the linear transformations in the VAE ensemble approach trivial transformations.
This measurement is also used in (Rolinek et al., 2019) for a similar purpose.
DtO is the Frobenius norm of the difference between a square matrix M and its closest signed permutation matrix
P ( M ) .
Finding P ( M ) can be formulated as a mixed-integer linear programming (MILP) problemas following:
[Equation]
By introducing new variables P + i,j , P − i,j ∈ { 0 , 1 } and D i,j = | M i,j − P ( M ) i,j | , we can reformulatethe above optimization problem as:
[Equation]
Using this optimization formulation, DoT of a given matrix M ∈ R n × n is deﬁned as:
[Equation]
C G EOMETRIC I NTERPRETATION OF THE E FFECT OF C ROSS - MODEL
[Figure] R ECONSTRUCTION
[Figure] Given a disentangled and entangled latent representation space, Fig. 5 illustrates the effect of thecross-model reconstruction by VAE ensemble.
The left part shows the orthogonal transformationfrom a disentangled representation to an entangled space, and the right part shows the transformation in the opposite direction.
As shown in the ﬁgure, the orthogonal transformation from thedisentangled representation to the entangled space does not introduce larger variance than the entangled representation.
Hence, we can expect similar cross-model reconstruction and within modelreconstruction.
However, the transformation from the entangled representation to the disentangledspace introduces larger variance (yellow shaded area over blue area on the right) than the disentangled representation.
This leads to larger cross-model reconstruction by the disentangled model.
D M ODEL A RCHITECTURE AND T RAINING D ETAILS
We conducted our experiments, including training and evaluating the current state-of-theart disentanglement models as well as evaluating the proposed VAE ensembles, using thedisentanglement lib 1 open-source library (Locatello et al., 2018).
Table 3 shows the encoder and the decoder architecture of the VAE model used in our experiments.
This architecture is the same as the one used in the original β -VAE Higgins et al.
Encoder
Decoder
Input 64 × 64 binary/RGB image
Input R d 4 × 4 conv, 32 ReLu, stride 2, pad FC d × 256, ReLu 4 × 4 conv, 32 ReLu, stride 2, pad4 × 4 upconv, 64 ReLu, stride4 × 4 conv, 64 ReLu, stride 2, pad4 × 4 conv, 64 ReLu, stride 2,
[Table]
Table 4 shows the hyperparameters setting used throughout the experiments.
These parameters areﬁxed for all the experiments.
E A DDITIONAL E XPERIMENTAL R ESULTS
In this section, we present the results for CelebA dataset similar to the ones conducted on dSpritedataset in Section 5.
In addition, we show the latent traversal of the trained model on both datasetsto further discuss the effect of VAE ensemble on the latent representation. [Table] [Figure]
E. C HARACTERIZATION OF THE L INEAR T RANSFORMATION IN VAE E NSEMBLE
In Figure 6, we show the DtO estimation of the linear transformations in the ensemble during training for the dSprite dataset. We report the mean and standard deviation of DtO across all lineartransformations over 10 different runs. Furthermore, we compare these results with a VAE baselinewhere DtO is estimated for the linear transformations between original VAEs. Speciﬁcally, we trainten different VAEs separately and estimate the DtO of the pairwise linear transformation amongthese models during training.
Similarly we report the mean and standard deviation of these DtO estimations. As seen in the ﬁgure, the VAE ensemble models with different ensemble size all approachto trivial transformations between the individual models, while the original VAEs do not have suchproperty. A similar result is also found in models trained for CelebA dataset. Similar to the resultsin Figure 6, we observe decreased DtO of the linear transformations in the VAE ensemble duringtraining.
We also compare models trained with different latent dimension size.
We observe decreased DtOas the latent dimension of the model increases inFigure 7. This is because, as discussed in themain paper, the VAE ensemble encourages more “active” latent variables. Models with higher latentdimension likely to learn a decomposition of generative factors. As a result, the alignment of thelatent variable between different models are easier thus the linear transformations between the latentrepresentations is closer to the trivial transformation.
On the contrary when there are less latentvariables in the model than the generative factors, some of the latent variables will capture morethan a single generative factor.
As a result, the one-to-one mapping between the latent variables ofdifferent models will not lead to a trivial transformation.
In Figure 9 and Figure 10, we show the “polarized regime” estimation for models in VAE E 2 and
VAE E 3 trained for CelebA datasets, respectively.
Similar to the results in Figure 3, individual models in the VAE ensemble tend to have similar ‘polarized regime”, and higher γ enforces the “polarized regime” by separating “passive” latent variables from the “active” ones.
When we comparebetween VAE E 2 and VAE E 3 , we observe increased “active” latent variables similar to the resulton dSprite dataset in Section 5.
More importantly, as discussed earlier, latent variables in a modelwith limited latent dimensions need to capture more than a single generative factor, especially fora complicated real-world dataset such as CelebA .
This makes the linear transformation between thelatent representations less trivial.
As the latent dimension size grows, such constraint is relaxed andthe linear transformations are closer to trivial.
These additional results conﬁrm the conclusion in Section 5: (1) as the ensemble size increases,
DtO increases due to the difﬁculty of aligning the latent representations among different models;(2) as the model latent dimension increases, DtO decreases due to the increased model capacity, andencourages the one-to-one mapping between latent variables in different models; (3) hyperparameterγ does not affect DtO signiﬁcantly, but plays an important role on separating “active” and “passive”latent variables, especially when the latent dimension is large enough.
Furthermore, we believe the DtO measurement of the linear transformation in VAE ensemble couldbe a useful indicator for latent dimension size.
As shown in Figure 7 and Figure 9, when the latentdimension is sufﬁcient for a given dataset, the DtO of the linear transformation is small and somelatent variables are pushed to “passive” mode. E. A BLATION S TUDY
Similar to the ablation result shown in Section 5, here we show the same ablation study using the
DCI Disentanglement metric.
Similar as the results of the FactorVAE metric in Fig. 4, removingeither component leads to a lower FactorVAE metric for the VAE ensemble. Without the linear transformation loss, the performance of VAE ensemble decreases signiﬁcantly across different ensemblesizes. E. L ATENT T RAVERSAL
In this section we show the latent traversal of models trained on both dSprites and CelebA datasets.
For a ﬁxed input image, to extract the latent traversal we change the value of a single latent variablez i in the corresponding encoding, and observe the generated output image to understand the effectof z i .
The range of the value are usually chosen to be from -3 to 3 due to the standard Gaussianprior.
In Figure 11, we show the latent traversal for both VAE E 2 and a single VAE model with 10 latentdimensions trained on dSprites dataset.
Three images as shown in the last column of each blockare used as input.
Both models are able to capture certain generative factors of the data includingposition, shape, rotation and scale.
In Section 5, we argue that the representation by VAE ensembleencourages more “active” latent variables, thus can capture a decomposition of the ground truth generative factors.
Especially from the “polarized regime” estimation in Figure 3, we observe that somelatent variables in the VAE ensemble are in-between “active” and “passive” modes.
This suggeststhat the VAE ensemble model generates input-dependent factors based on the input complexity.
In
Figure 11, we observe this behavior highlighted with color boxes.
The traversal on the second latent variable z 2 shows that an ellipse shape does not lead to an “active” latent variable.
However,both heart and square shape lead to an “active” latent variable that changes the output.
In contrast,the single VAE model does not have such behavior where the “active” modes are consistent acrossdifferent input data.
In Figure 12 and Figure 13, we show the latent traversal for both VAE E 2 and a single VAE modelwith 16 latent dimensions trained on CelebA dataset, respectively.
In this real-world dataset, thegenerative factors are unknown.
We observe different factors including background, azimuth, gender, hair style being captured by both models.
Similar as before, the single VAE model maintainssimilar “active” mode for all latent variables where similar traversal patterns are observed for bothinput images.
However, VAE E 2 shows semantically consistent but input-dependent “active” mode.
This is translated into different traversal effects and more realistic and sharper images by VAE E 2 ,especially for the ﬁrst input image that is less common in the dataset.
We believe this is importanttowards a meaningful compositional latent representation learning.
Overall, the latent traversal results in this section conﬁrm the ﬁndings on the inner working of the
VAE ensemble shown in the previous section as well as the discussion in Section 5.
[Figure]
[Figure] [Figure] [Figure]