W HAT
CANNOT
BE
LEARNT BY V ARIATIONAL A UTOENCODER ?
Anonymous authors
Paper under double-blind review
A BSTRACT
What would be learnt variational autoencoder(VAE) and what inﬂuence the disentanglement of VAE?
This paper tries to preliminarily address those issues theoretically in the idealistic situation and practically through noise modeling perspective in the realistic case.
Information conservation (e.g. 2 Gaussian and 3 Gaussian cannot be the generating factors of each other), factor equivalence (e.g. the orthogonal transformation of Gaussian factors is still Gaussian factors) and separation of mutual information properties in the idealistic VAE case are revealed.
They preliminary but fundamentally answer what would be learned by idealistic VAE and what cannot.
Several performance indicators regarding the disentanglement and generating inﬂuence are subsequently raised to evaluate the performance of VAE model and to supervise the used factors.
On other fold, the implementations under noise modelling empirically testify the theoretical analysis and also show their own characteristic in pursuing disentanglement.
I NTRODUCTION
Variational AutoEncoder(VAE)s (Kingma & Welling (2013), Rezende et al.
(2014)) have shown their powerful human-like abilities: modelling causal relationship, unsupervisedly extracting disentangled factors/representation (Bengio et al.
and generating signals with abundant diversities in a “latent-factor-controllable” way.
Those capabilities enable the knowledge transferring through shared causes/factors among different tasks/experiences, emphasized as the important human advantages against the current machine by Lake et al.
and compling with the ideal mental imagery mechanism in memory and thinking.
Beneﬁtted from those capabilities, VAEs have been widely applied to various applications, including disentangled representations learning of images and time series (Higgins et al.
(2016), Kulkarni et al.
(2015), Mathieu et al.
(2016), (Fabius & van Amersfoort (2014)
However, the lack of public theoretical study regarding the generating and inference procedure induced by VAEs is tripping the research process: we wouldn’t exactly know whether the VAE could learn the real factors or just some fantasies that model itself makes up; we couldn’t exactly clarify the need and range induced by the word “disentanglement”; we actually couldn’t exactly guarantee the effectiveness of the current disentanglement metric.
All those lacks distillation would make researchers hard to effectively compute through their knowledge from the experiments and incline to make some avoidable arguments and considerations.
This paper will ﬁrst discuss the properties of the idealistic VAE, that is, roughly the VAE whose decoding procedure and encoding procedure are both deterministic.
The information conservation theorem, the clariﬁcation on disentanglement, factor equivalence class, separation of mutual information property and several indicators will be introduced.
After that we relax the discussion to the case that decoding procedure are not deterministic.
The experiments are design to testify whether the knowledge derived form the idealistic case would be applied to the realistic sampling case and to study the behavior of different noise assumptions as well as our indictors. [Figure]
G AUSSIAN -VAE M ODEL
Gaussian-VAE (Kingma & Welling (2013), Rezende et al.
(2014)) is an scalable unsupervised representation learning model (Higgins et al.
and since Gaussian distribution can be continuously and reversibly mapping to many other distributions, the theoretical analysis on it is also instructive for other continuous latent factors VAE.
Gaussian-VAE assume that input x is generated by several independent Gaussian factors z , that is p θ ( z ) =
N ( z | 0 , I H ) .
The generating/decoding process is modeled as p θ ( x | z ) and the inference/encoding process q φ ( z | x ) is treated as the approximate posterior distribution.
Both of them are parameterized by the neural network with parameter θ and φ .
In VAE setting, the approximate inference method is applied to maximizing the variational lower bound of p
[Equation]
=
[Equation]
G ENERATING F ACTOR P ROPERTIES AND P ERFORMANCE I NDICTORS
In order to get assess to the question, whether the VAE could learn the real factors or just some fantasies that model itself makes up, we start the analysis from the idealistic situation 1 .
3. D ETERMINISTIC A SSUMPTION ON q φ ( x | z ) AND p θ ( z | x )
If the factors of x are well understood, the generation process should be deterministic, that is p θ ( x | z )
= δ ( x = ψ ( z )) .
We limit the consideration that q φ ( z | x ) =
δ ( z = ψ − 1 ( x )) is also deterministic as well for simplicity of analysis in this paper.
For more complex situation, this consideration could be basic and instructive.
3. I NFORMATION C ONSERVATION
We try to address this issue by disregarding the training procedure and direct considering the idealistic VAE’s behavior through the following theorem.
Theorem 1 (Information Conservation) .
Suppose that z = ( z 1 , · · · , z H ) and y
=
( y 1 , · · · , y P ) are sets of H and P ( H (cid:54)
= P ) independent unit Gaussian random variables, respectively, then these two sets of random variables can not be the generating factor of each other.
That is, there are no continuous functions f : R H
→ R P and g : R P → R H such that z
= g ( y ) and y = f ( z ) .
Proof in Appendix 7.3.
The principle of the theorem is visually illustrated in Fig.
3.2. E FFICIENT R EPRESENTATION AND C LARIFICATION ON D ISENTANGLEMENT
According to the information conservation theorem, the independent unit Gaussian factor assumption forms a strong inductive bias and facilitates the model incline to achieve most efﬁcient coding.
Under this assumption, the number of the learnt “used” factors of idealistic VAE should be the same as the true factors number under some assumptions such as the learnt q φ ( z ) should equal p θ ( z ) and decode/encode process is continuous and reversible.
Empirically, sometimes only a small amount of unit Gaussian variables regarding the factors of VAE have been used and the q φ ( z | x ) is close to deterministic though the number latent of factors sometimes is pre-speciﬁed larger.
This theorem helps provide an interpretation to explain this phenomenon.
Here, in order to avoid the ambiguity of the terminology of disentanglement in this paper, based on the information conservation theorem, we make the following clariﬁcation.
• The disentanglement of the learnt representation/factors in this literature refers to two parts depicted in Theorem 1: – the factors are closer to be independent with each other, – the factors incline to be able to generate the oracle signal and to be inferred perfectly from the oracle signal through a continuous procedure/mapping.
• The “disentanglement” refers to the closeness of the learnt factors to the pre-speciﬁed independent factors/concpets that can generate the oracle signal and be perfect inferred through a continuous procedure/mapping such as the independent semantic/visual factors.
Therefore the estimation for D KL ( q
φ
( z ) |
|
p θ ( z )) that reﬂects the divergence of
the learnt factor distribution and the i.i. d. unit Gaussian prior can be good a indicator to supervise the independence of the factors and served to quantitatively assess the disentanglement of each extracted factor, while the similarity regarding the original signal and reconstruction place another part of the disentanglement.
The “disentanglement” will be shown hard to be obtained in an unsupervised manner.
Concretely, even in the idealistic cases, the extracted factors tend to possess the intrinsic number of latent factors of the model, while there are still possibly large variations of these factors due to it can be obtained only in the equivalent class induced by the pre-speciﬁed factors as proved in the next subsection.
3. F ACTORS E QUIVALENCE : W E SHOULD NOT EXPECT “ ONE - TO - ONE ” CORRESPONDENCE BY DISENTANGLEMENT .
Theorem 2 (Gaussian Factor Equivalence) .
Suppose that z = ( z 1 , · · · , z H ) is a set of H independent unit Gaussian random variables.
Let Q ∈ R H
× H be an orthogonal matrix and then y =
Qz is also a set of H independent unit Gaussian random variables.
Besides, z and y can generate each other through a linear homeomorphism mapping.
Proof in Appendix 7.3.
This theorem implies that there are a class of unit Gaussian random variables which can generate each other and have equivalent conservation information, as indicated by the following theorem.
Theorem 3 (Linear Gaussian Factor Equivalence Class) .
[Equation]
[Figure] Then ∀ y ∈ [ z ] , y is a set of H independent unit Gaussian random variables and can generate z through an linear homeomorphism mapping.
The theorem clariﬁes that if Gaussian-VAEs have an linear matrix multiplication freedom degree of learning the factors, then the factors in the equivalence class can all be possibly learnt.
The empirically results tally with the above analysis(see Fig. 2).
Suppose the visual semantic concepts can be viewed as a set of independent Gaussian variables ( z = ( z rotation , z gender , z with − glass , · · · ) T ) which are desired to be captured and learnt by VAEs, while the model is also possible to learn the independent factor set y
=
( y 1 , · · · ) T =
Qz in the equivalence class [ z ] .
This explains why changing one factor like y 1 sometimes empirically results in change in multiple visual concepts.
This perspective suggests that it’s actually hard to obtain the “disentangled representation” that exactly “one-to-one” corresponds to the “independent semantic representation” even though they are in the same equivalence class.
As a result, the idealistic VAE model just tend to learn the “entangle representation” if we do preset “oracle generating factors” belonging to the equivalent class.
However, though those conclusions might be upsetting, it seems not be biology impossible.
A neuron in hippocampus of animals was suggested to combinatorially possess several representation capabilities.
E.g., Aronov et al.
(2017) found that some neurons in rat’s hippocampus involved in spatial representation also were involved in representing sound frequencies after training rats by a tasks that required them to use a joystick to manipulate sound in frequency continuously.
3.3. D EFICIENCY OF THE E XISTING D ISENTANGLEMENT M ETIC
Higgins et al.
(2016) proposed a “simulated factor” based “disentanglement” metric on the simulation datasets.
However, according to Gaussian factor equivalence theorem 2 that even idealistic
VAE will still learn the factors in the equivalence class, their metric could be effective sometimes for disentanglement but might suffer instability when evaluating the VAE in different trials (detailed in Appendix 7.4).
Further, this metric could be hardly calculated in the real datasets to provide direct feedback of the “disentanglement”.
The reason is that it must pre-know the generating factors expected to be learnt.
3. S EPARATION OF THE M UTUAL I NFORMATION
The mutual information regarding the factors learned by the inference/encoder network and the signal x can be a good quantity for evaluating the generating inﬂuence.
That is,
[Equation]
In order to understand and estimate which factor of the VAE was learnt and inﬂuenced the generating process, I encoder ( x ; z h ) can be taken as a rational indicator 3 .
If we assume that z 1 , z 2 , · · · , z H is conditional independent given x 4 , it can yield a useful result as the following.
Theorem 4 (Separation of the Mutual Information) .
Let z 1 , · · · , z H be independent unit Gaussian distribution, and z 1 , z 2 , · · · , z H be conditional independent given x .
Then
[Equation]
Proof in Appendix 7.3.
This theorem suggests that if the learnt q φ ( z ) can factorize and the q φ ( z | x ) can factorize, then the consideration of each I encoder ( z h ; x ) won’t be excess or lose information.
Besides, when those term comes in the optimization objective, then it can start the negotiation between the information preservation and dimension reduction and play the role of lasso that clip down the factor in dimension with small mutual information.
Theorem 5.
The terminology follows the aforemention deﬁnitions and if the involved KL-divergence and mutual information is well deﬁned, then
[Equation]
Proof in Appendix 7.3.
The theorem demonstrates that the second term in variation lower bound in
Eq.
(11) is capable of controlling both the mutual information of x and z induced by the encoder network as well as the similarity of the learnt q φ ( z ) and the prior p θ ( z ) .
Further, the theorem suggests that it possesses the lasso capacity of clipping down the non-intrinsic factor dimension to some extent.
3. I NDICATORS
In order to quantify the disentanglement performance 5 as well as the I encoder ( x ; z ) , we assume that q ∗ ( z ) is a factorized zero mean Gaussian estimation for q φ ( z ) .
We can then list the indicators for assessing latent factor disentanglement:
Deﬁnition
[Equation]
Deﬁnition 2 (Estimation for I encoder ( x ; z ) ) .
[Equation]
[Equation]
Visually recognition could also provide a assess equivalent supervise disentanglement.
If we assume that the most of the visual concept/factors follow the assumption regarding the “disentanglement”, it is rational to qualitatively measure the interpretability of extracted latent factors by human perception to infer the disentanglement.
Besides, previous empirical evidences of VAE applications (Higgins et al.
(2016), way
Deﬁnition 3 (Estimation for I encoder ( x ; z h ) which quantiﬁes the inﬂuence of each factor) .
[Equation]
Deﬁnition 4 (Estimation for D KL ( q φ ( z ) || p
θ ( z )) ) .
[Equation]
Note that the above indicators 2-4 need the value of q ∗ ( z ) , we now introduce how to calculate this term based on Theorem 5.
Through the minimization equivalence, we know that
[Equation]
the q ∗ ( z ) can then be obtained by gradient method from solving the following optimization problem.
[Equation]
VAE WITH N OISE M ODELING
In order to testify the idealistic consideration in real situation, that datasets have noise 6 (this situation correspond to that p θ ( x | z ) is not deterministic since the major factors z only forms a subset of the whole factors.
7 ), we integrate the noise modeling into our model:
[Equation]
where σ 2 is either manually enumerated or adaptive learned.
Noise modeling can be found crucial in inﬂuencing disentanglement in experiment since it would actually deﬁne the factors learnt and subsequently inﬂuence the learnt intrinsic dimension suggested by information conservation property of VAE.
4. N OISE M ODELLING W ITH A UXILIARY C ONSTRAINT
The entangled representation can be caused by the over-large of searching space of q φ ( z | x ) .
If the learned q φ ( z ) = (cid:82) q φ ( z | x ) p data ( x ) dx has a big divergence to p θ ( z ) , then the VAE model tends to learn the entangle representation as it violates the one part of the disentanglement (clariﬁed in section 3.2.1).
Actually, in the VAE model, what we want is to search in the space that q φ ( z ) is possibly similar to p θ ( z ) 8 .
By implementing this ideal, we add auxiliary upper bound E
[Equation]
( q φ ( z | x ) || p θ ( z )) (detailed in Theorem 5) of D KL ( q φ ( z ) ||
p θ ( z )) to the original objective.
This equivalently leads to the approach of β -VAE raised by Higgins et al.
[Equation]
where β > 1 .
and 4.1. R ELATION OF VAE AND β -VAE UNDER G AUSSIAN NOISE ASSUMPTION WHEN σ 2 IS PRE - SPECIFIED Equivalent objective of σ 2 pre-speciﬁed Gaussian noise VAE: [Equation]
Equivalent objective of σ 2 pre-speciﬁed as σ 2 pre Gaussian noise β -VAE:
[Equation]
where we call βσ 2 pre the normalized variance.
It’s shown that when the σ 2 is pre-speciﬁed, manually tuning it is the same as manually tuning β with a ﬁxed σ 2 pre (for example σ 2 pre = 1 ) under Gaussian noise assumption.
This equivalence saves our time for extra experiment studying the behaviors of this two cases and we call those two case noise speciﬁed β -VAE.
E XPERIMENT
The experiments will try to give the answer to the following questions:
• Would noise modeling/speciﬁcation inﬂuence the disentanglement?
• What’s the difference characteristic of disentanglement induced by auxiliary constraints?
• Would the information conservation theorem be instructive in real implementation?
• Would the generating factor equivalence class properties exist in real implementation?
• How do our indictors act and whether they can determine the used the factors or not?
5. D ATASET
MNIST is a database of handwritten digits (Lcun et al.
CelebA (Liu et al. (2015)) is a largescale celebfaces attributes datasets and only its images are used in our experiments.
More details are in Appendix 7.5.
The extensive comparison of Gaussian-noise modeling β -VAE and Gaussiannoise with speciﬁed variance ( β )-VAE is made on MNIST based on our indicators to exploring the disentanglement as well as to testify the theorem and analysis from the idealistic case to the realistic sampling case.
The experiments on CelebA will be auxiliary to further support the generating factor equivalence class theorem 9 .
By setting β as different values, we compare the performance of β -VAE with and without prespeciﬁed noise on MNIST.
We speciﬁcally listed the result of β (=1)-VAE in all cases.
More details can be found in Appendix 7.5.2.
5. E XPERIMENTS ON FOR Q UESTIONS
• Noise modeling/speciﬁcation inﬂuence the disentanglement
The noise speciﬁcations and modeling signiﬁcantly inﬂuence the model evidence quantitatively and reconstruction qualitatively, as clearly shown in Fig.
The noise speciﬁcations signiﬁcantly inﬂuence the divergence regarding q φ ( z ) and p θ ( z ) and noise speciﬁed and noise learning β -VAE achieve similar disentanglement quantitatively based on the similar indicator behaviours of ˜ D KL ( q φ ( z ) ||
p θ ( z )) in Fig.
(4a) and the number of normal variance factors Fig.
(4b) in regard to the normalized variance.
[Figure]
E m p
i r i c a l V a r i a t i o n a l L o w e r B
[Figure]
u n
[Figure]
[Figure]
t i m
[Figure]
t ed D K L ( q ( z ) || p ( z )) ( na t s )
[Figure]
• Difference induced by auxiliary constraints is found. The prominent difference induced by auxiliary constraints would be its stronger suppression on ˜ I encoder ( x ; z ) and the number of inﬂuential factors although changing the noise level also possess this capability indirectly.
It’s somewhat obvious to see and compare value of different indictors under the different β setting by sliding on the green/blue line.
The bigger β , the lower ˜ D KL ( q φ ( z ) ||
p
θ ( z )) and roughly the better reconstruction and hypothesis learnt.
However, it’s more interesting that in regard to the normalized variance, noise learning β -VAE enhances the suppression on ˜ I encoder ( x ; z ) , as depicted in Fig.
(5a)
and that is comprehensible since β -VAE is minimizing the auxiliary constraints both I encoder ( x ; z ) + D KL ( q ( z )
|| p ( z ))
based on Theorem 5.
• ˜ I encoder ( x ; z h ) effectively determines the “used” factors and information conservation hold in sampling case when the disentanglement achieves in some extent.
As shown in Fig.
(6) and (16), the indicator ˜
I encoder ( x ; z h ) determines the “used” factors.
According to those ﬁgures, VAEs automatically suppress the auxiliary factors and learn the factor dimension as [Figure] [Figure]
β VAE ( β = 10 , equivalent σ=
The montages: inﬂuential factor traversals.
We select those factor traversals with visually most interpretable/comprehensive effects to present and the whole inﬂuential factor traversals are listed in appendix 13.
The phenomenon of the multiple semantic change induced by the same learnt factor and the encoding of same semantic among different learnt factor tallies with factor equivalence class theorem 2.
The similar plot of its counterpart with speciﬁed normalized noise can be found in Fig.(16) in Appendix.
it still capable to have good reconstruct abilities and its D KL ( q φ ( z ) || p
θ ( z )) closer to zero under the suitable noise assumption.
This phenomenon has already suggested by the information conservation theorem 1.
• Factor equivalence is generally hold.
The reﬂection of the generating equivalence properties 3.3, that is, single factor could result in multiple semantic concepts change and same semantic concept could be encoded in different factors, are again well demonstrated by Fig.
(7) and Table 1.
P ERSPECTIVE
From the perspective of representation learning:
• It is interesting that the topology properties of oracle signal are used to obtain the proof for the information conservation theorem.
Other situations including that data owns several connected components can be further considered and would uncover the efﬁcient coding properties of discrete factors.
[Table] • When q φ ( z | x ) is far from deterministic, the discussion would be crucial for many general purposes induced by word disentanglement.
Those study may further extend to the case that data containing different dimension manifolds and to the core VAEs’ pattern separation/completion/generalization capabilities.
A PPENDIX 7. G AUSSIAN N OISE D EDUCTION
The VAE objective (i e., the variational lower bound) can be treated as function of the θ and φ where noise variance parameters σ 2 are contained in θ ,
[Equation]
Here the
SGVB estimator in (Kingma &
Welling
[Equation]
[Equation]
[Equation]
is used.
Note that the noise variance σis also taken as an optimization variable in the model, making the model capable of better adapting noise variation of data in practical cases in a totally automatic way, instead of a manually set manner.
Given multiple data points from a dataset X, we can construct an estimator of the mean marginal likelihood lower bound of the full dataset, based on minibatches
[Equation]
where the minibatch X M = { x m } Mm =1 is a randomly drawn sample set of M datapoints from the full dataset X. Such a lower bound also constitutes an important indicator for model evidence in latter experiment.
We call it the empirical variational lower bound (EVLB) in the following.
Note that ˜ L B ( θ, φ, x m ) (cid:39) L ( θ, φ, x m ) and we can deduce that
[Equation]
The last inequality holds due to D KL ( p data ( x ) || p θ ( x )) ≥ 0 .
The alternative optimization strategy can be readily utilized to design the algorithm for solving the model by iteratively updating the noise parameter and the network ones.
During the optimization process, the objective can be monotonically increasing, and thus the algorithm can be guaranteed to be convergent.
The algorithm is summarized as follows:
Optimization for parameters except σ 2 :
Optimization
[Equation]
(Close form solution in regard to ˜ L M ( θ [Equation] φ, X M ) .)
Direct gradient method to the transformed variable log simga = log σ ∈ R can be implemented to lift the lower bound ˜
L M as a result to increase the likelihood as well.
7. M O G N OISE D EDUCTION
The noise ε in real situation might be more complex than a simple Gaussian, like that existed in real photographs (Plotz & Roth (2017)).
We thus try to further ameliorate the noise setting as a mixture of Gaussian(MoG) noise.
Such noise modeling strategy has been widely veriﬁed to be effective in applications, like matrix factorization (Meng & Torre (2014)) and robust principal component analysis (Zhao et al.
That is, we assume that
[Equation]
Let c d ∈ { 0 , 1 } K be the latent indicator random one-hot variable, (cid:80)
Kk =1 c dk = 1 , for the MoGnoise component of pixel indexed by d .
Let Π =
[ π 1 , · · · , π K ] and Σ =
[ σ 21 , · · · , σ 2 K ] be the ratio and variance of each component, respectively.
Let Π , Σ be contained in θ .
The conditional joint distribution turns to be
[Equation]
The posterior distribution q φ ( z, c | x ) can be factorized as q φ ( z | x ) q ( c | x, z ) , where q φ ( z | x ) will be direct learnt and the alternative of q ( c | x, z ) , q ( c | x, e ) will be set to the last step p θ ( c | x, e ) in regard to EM procedure.
The lower bound of log p θ ( x ) is then reformulated as follows:
[Equation]
Similar to the Gaussian case, the reparamerization trick is implemented,
[Equation]
18) where ˜ z =
En ( x ) + Σ z | x 1 / 2 ( x ) e .
By utilizing the SGVB estimator, we get,
[Equation]
Given an input dataset X, we can then construct an estimator to the mean marginal likelihood lower bound of the full dataset, based on minibatches, as follows:
[Equation]
where z m,l =
En ( x m ) + Σ z | x 1 / 2 ( x m ) e ( l ) and the minibatch X M = { x m } Mi =1 is a randomly drawn sample of M datapoints from the full dataset X.
Then let
[Equation]
where z old =
En old ( x ) + Σ oldz | x 1 / 2 ( x ) e , and we can get
[Equation]
The EM algorithm can be naturally employed to solve the model.
The implementation steps are listed as follows:
Step 1.
Expectation Step .
Set q ( c | x m , [Equation] ( l ) ) =
p oldθ old ( c | x m , En old (
[Equation]
oldz |
[Equation]
( x
[Equation]
[Equation] [Equation] [Equation] m , l = 1 , · · · , L .
Calculate the expectation of the latent variable c :
[Equation]
where z m,l : z m,lold =
En old ( x m ) + Σ oldz | x ( x m ) e ( l ) .
The Objective in Maximization Step is obtained as the following,
[Equation]
Step 2.
Maximization Step:
Fix: q ( c | x, e ) determined in the Expectation Step. M
[Equation]
Update [Π , Σ] and { θ, φ } / [Π , Σ] by alternative optimization strategy.
Update Π , Σ : note here z m,l : z m,lold =
En old ( x m ) + Σ oldz | x ( x m ) e ( l ) , and we can easily get the closed-form updating formula for these parameters:
[Equation]
Update
{ θ, φ } / [Π , Σ] : gradient methods with respect to { θ, φ } / [Π , Σ] .
Note here z m,l = En ( x m )+ Σ z | x ( x m ) e ( l ) .
The algorithm can then be summarized as follows:
Initialize the coefﬁcient of { θ, φ } / [Π , Σ] and the coefﬁcient of noise
ε : Π , Σ Sample e from N (0 , I H ) to obtain e 1 , · · · , e M [One for each element sample in the mini batch in the next step ( L here is set to 1)].
Sample a mini batch X M from p data ( x ) .
Implement EM algorithms as aforementioned (approximate inference for q ( c, z | x ) ): Expectation: calculate γ dmk .
Maximization: update [Π , Σ] , Update { θ, φ } / [Π , Σ] with gradient methods. 5.
Goto 3: Until Trigger End-Criterion.
7. P ROOF
Proof.
For theorem 1.
Proof by Contradiction.
Suppose those two function exist, and we will show that they will be inverse mapping of each other and the homeomorphism mapping of R H and R P .
Since R H and R P have different topology structures ( P (cid:54) = H ), the homeomorphism mapping will not exist.
[Equation]
Since both f and g are continuous, there is a homeomorphism mapping between R H and R P and it leads to the contradiction.
Proof.
For theorem 2.
We only need to test the mean and variance of y .
[Equation]
Therefore, y is another set of H independent unit Gaussian random variables.
Since x = Q T y , z
and y can generate each other with an linear homeomorphism mapping.
Proof.
For theorem 4,
[Equation]
Proof.
For theorem
[Equation]
7.3. A UXILIARY E XPLANATIONS FOR I NDICATORS
Corollary 1.
The terminology follows the aforemention deﬁnitions and if the involved KLdivergence and mutual information be well deﬁned then
[Equation]
The proof of corollary 1 is the same as that of theorem 5.
This corollary suggests that the estimation in deﬁnition 2 provides another upper bound for the capacity of the encoder network.
Empirically, this estimation is a much tighter estimation than using the estimation in deﬁnition 1.
Corollary 2.
The terminology follows the aforemention deﬁnitions and if the involved KLdivergence and mutual information be well deﬁned then
[Equation]
The corollary is an direct result of theorem 5 and corollary 1.
It suggests that the estimation in deﬁnition 4 is a lower bound for D KL ( q φ ( z )
|| p θ ( z )) .
Deﬁnition 5 (Another Estimation for D KL ( q φ ( z ) ||
p θ ( z )) ) .
[Equation]
Empirically, ¯ D KL ( q φ ( z ) ||
p θ ( z )) and ˜ D KL ( q φ ( z ) ||
p θ ( z )) shown the same estimation results on MNIST.
Deﬁnition 6 (Another estimation for I encoder ( x ; z ) ) .
[Equation]
Deﬁnition 7 (Another estimation for I encoder ( x ; z h ) which quantiﬁes the inﬂuence of each factor) .
[Equation]
7. A NALYSIS ON THE “D ISENTANGLEMENT ” M ETRIC RAISED IN β -VAE (H IGGINS ET AL . (2016))
The terminology inherits those in the β -VAE paper.
The main idea of that “disentanglement” metric is to create a statistic point z diff relevant to the model for each simulated factor respectively and then to use a linear classiﬁer to project the statistic point to the corresponding index of the simulated factor.
If the statistic points induced by the model are easy to be separated then the model is thought to learn “disentangled” representation.
Here, we will argue that even for the idealistic VAE model that follows the disentanglement conditons 3.2.1 could still receive bad score under that performance metric in some situations.
Suppose that the true simulated factors v follows N (0 , I H ) .
Then the learnt “used” 10 factor z can be in the equivalence class [ v ] according to theorem 2.
Concretely, there exists an orthogonal transformation Q such that z
=
Qv .
Suppose that the simulated factors with index y of v is ﬁxed.
Suppose v 1 y − fixed and v 2 y − fixed are two random variable representing the samples from the y -ﬁxed v .
Then the factors inferred by the idealistic VAE turns to be z
1 =
Qv 1 y − fixed and z 2 =
Qv 2 y − fixed .
In order to calculate the statistic point z diff ( y ) =
E | z 1 − z 2 | , we ﬁrst calculate the mean and variance of ( z 1 − z 2 ) .
[Equation]
V ar [Equation]
y − fixed − v 2 y − fixed ) [Equation]
=
QCov [Equation] v 1 y − fixed − v 2 y − fixed , v 1 y − fixed − v 2 y − fixed )[-Q T = Qdiag (2 , · · · , 2 , 
-]
Q T [-= 
-][Equation] y
[Equation]
[Equation]
Therefore, z diff ( y ) can be obtained through the diagonal value of 2 I − 2 q y q Ty .
That is,
[Equation]
From the above equation, the location of statistic point is unique determined by ( q 2 y 1 , · · · , q 2 yH ) .
When ( q 2 y 1 , · · · , q 2 yH ) is close to the vertex of the unit cubic for each y then all the statistic points turn to be easily separated.
However, from the perspective of the objective, all the orthogonal Q s are with the same potential to be learnt.
It seems not to be with a small probability that statistic points of different indexes take similar location.
For instance, if H = 2 and Q = (cid:32)
[Equation]
(cid:33) then z diff (1) =
z diff (2) cannot be separated while the representation still follows the disentanglement conditions.
Among different trials, the Q might contribute to that “disentanglement” metric but also might not.
That explains why that metric is unstable.
7. E XPERIMENT D ETAILS
We set L to 1, and minibatch size M to be 100 in all practices.
All the pixels value have been linear normalized in to [0,1].
7.5. C LARIFICATION ON THE CORRECTION ON RESULTS
We ﬁnd in the last version that the code on calculating q ∗ ( z ) is wrong.
Concretely, the objective of the KL-divergence of two Gaussian is incorrect calculated.
That inﬂuences the estimation of ˜
I encoder ( x ; z ) , ˜ I encoder ( x ; z h ) , ˜ D KL ( q φ ( z ) || p
θ ( z )) and σ 2 z H but has little inﬂuence on the determination of the “used” factors.
So we redo all the experiments on MNIST and delete the relevant results regarding those wrong-calculated indictors on CelebA and Extended Yale Face B.
The new experiment results solve many our past confusions due to the wrong experiment.
They are
• Why σ 2 z h is strongly correlated with ˜ I
( x ; z h ) ?
• Why σ 2 z h of the used factor is always relative small?
• Why can VAE still learn the “disentangled” representation when the learnt ˜ D KL ( q φ ( z ) ||
p θ ( z )) is such big?
Now we know that they are actually not the cases.
Something better is that we ﬁnd our experiment results are more close to our theoretical analysis: when guaranteeing the reconstruction quality in a tolerance range, the smaller ˜ D KL ( q φ ( z ) ||
p θ ( z ))
the less “used” factors are learnt.
That coincide with information conservation theorem: the independent unit Gaussian of the factors assumption facilitates the most efﬁcient coding.
7.5. M NIST
We split randomly 7000 datapoints according to ratio [0 . 6 : 0 . 2 : 0 . 2] into training set, validation set (no use), testing set.
All the indicators and q ∗ ( z ) are evaluated/calculated on 10000 datapoints belonging to the testing set.
All the seed images used to infer latent code and to draw the traversal come from the testing set.
In all ﬁgures of latent code traversal each block corresponds to the traversal of a single latent variable while keeping others ﬁxed to either their inferred ( β -VAE, seed image used to infer the latent values in the VAE-based models.
β -VAE and VAE traversal is over the [ − 3 , 3] range.
The assumed variance σ 2 of noise speciﬁed Gaussian of VAE models is enumerated from [0 . 0005 , 0 . 001 : 0 .
The β setting for the noise learning β -VAE is enumerated from [0 . 1 , 0 . 5 , 1 , 2 : 2 : 18] .
[Figure] [Figure]
[Figure] [Figure] [Figure] [Table]
We split randomly 2424 datapoints according to ratio [0 . 8 : 0 . 1 : 0 . 1] into training set, validation set (no use), testing set.
The model is training on the training set.
All the seed images used to infer latent code and to draw the traversal come from the 100 datapoints from the testing set.
In all ﬁgures of latent code traversal each block corresponds to the traversal of a single variable while keeping others ﬁxed to either their inferred β
VAE, different seed image used values based β -VAE and VAE traversal is over the [ − 3 , 3] range.
The β setting for the noise learning β -VAE is enumerated from [
7.5. C ELEB A
We split randomly roughly 200000 datapoints according to ratio [0 . 8 : 0 . 1 : 0 . 1] into training set, validation set (no use), testing set.
The model is training on the training set.
All the indicators and q ∗ ( z ) are evaluated/calculated on 10000 datapoints selected from testing set.
All the seed images used to infer latent code and to draw the traversal come from the 100 datapoints from the testing set.
In all ﬁgures of latent code traversal each block corresponds to the traversal of a single latent variable while keeping others ﬁxed to either their inferred ( β -VAE, VAE).
Each row represents a different seed image used to infer the latent values in the VAE-based models.
β -VAE and VAE traversal is over the [ − 3 , 3] range.
The β setting for the noise learning β -VAE is enumerated from [1 , 30 , 40] .
7.5. N ETWORK S TRUCTURE
[Table]
7. A UXILIARY G ENERATING P ICTURE
Note that only the factors with ˜ I encoder ( x ; z h ) > 0 . 5 are shown.
VAE was proposed by Kingma & Welling (2013) and Rezende et al. (2014) to implement the efﬁcient learning and inference in directed probabilistic models regarding continuous latent variables with intractable posterior distributions and in scalable datasets. They introduced a network inference/recoginition model to represent the approximate posterior distribution and utilized reparameterization trick for stochastic joint optimization of a variational lower bound containing the parameters of both the generative/decoder and inference/recoginition/encoder models.
After being raised, many VAE variations have been proposed to boost VAE’s capabilities in generation quality and/or disentanglement of the learned representation. In these methods, multiple efforts were made by improving the generative and inference network structures.
Typical works along this line include the convolution/de-convolution structure raised by Kulkarni et al. and ladder structure raised by Zhao et al. Some other works advanced the mechanism under the VAE generation/inference processes. Typical works include the iterative attention generation/inference
[Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure]
Despite the improvement to the VAE itself, some other efforts were made by the ensemble between GAN with VAE. E.g., Larsen et al. (2015) uniﬁed GAN and VAE to obtain a better reconstruction and a high-level abstracts visual features embedding.
Mathieu et al. (2016) also uniﬁed GAN and VAE but put emphasis on disentangling factors of variation. GANs without auxiliary design would
[Figure]
learn the data distribution disregarding its noise level though suffer from unstable training and mode collapsing (Salimans et al. (2016)) while VAEs would assume a decomposition of the noise and oracle clean datapoint regarding the noise data with an auxiliary prior on the distribution regarding the factors. Besides, many efforts were made by regularization on the factor distribution or factor generating effect.
E.g., Makhzani et al.
(2015) introduced an adversarial loss into the latent space of the autoencoder which in idealistic case could learn any kind factor/lantent distribution including those contributing to the disentangled factors/representation. InfoGAN, raised by Chen et al.
[-Gaussian 1 positive σ 22 = 0 . 
-](2016), introduced the infomax principle to GAN by adding an auxiliary mutual information regularization which enabled the inference of GANs and led to a better disentangled representation as well. Recently, there is a new VAE variation is proposed by Higgins et al. (2016) who introduced the β -VAE framework which enhanced the constraints regarding the KL-divergence of the posterior and prior distribution of VAE and showd a novel disentanglement performance. This method has obtained a better performance as compared with conventional VAE methods, especially on its ﬂexible tuning a compromising a parameter beta between the KL-divergence term and the likelihood term (the variational lower bound).
[Figure]
[Figure] [Figure] n
[Figure]