P RELIMINARY THEORETICAL TROUBLESHOOTING IN V ARIATIONAL A UTOENCODER
Anonymous authors
Paper under double-blind review
A BSTRACT
What would be learned by variational autoencoder(VAE) and what inﬂuence the disentanglement of VAE?
This paper tries to preliminarily address VAE’s intrinsic dimension, real factor, disentanglement and indicator issues theoretically in the idealistic situation and implementation issue practically through noise modeling perspective in the realistic case.
On intrinsic dimension issue, due to information conservation, the idealistic VAE learns and only learns intrinsic factor dimension.
Besides, suggested by mutual information separation property, the constraint induced by Gaussian prior to the VAE objective encourages the information sparsity in dimension.
On disentanglement issue, subsequently, inspired by information conservation theorem the clariﬁcation on disentanglement in this paper is made.
On real factor issue, due to factor equivalence, the idealistic VAE possibly learns any factor set in the equivalence class.
On indicator issue, the behavior of current disentanglement metric is discussed, and several performance indicators regarding the disentanglement and generating inﬂuence are subsequently raised to evaluate the performance of VAE model and to supervise the used factors.
On implementation issue, the experiments under noise modeling and constraints empirically testify the theoretical analysis and also show their own characteristic in pursuing disentanglement.
I NTRODUCTION
Variational AutoEncoder(VAE)s (Kingma & Welling (2013), Rezende et al.
(2014)) have shown their powerful human-like abilities: modelling causal relationship, unsupervisedly extracting disentangled factors/representation (Bengio et al.
and generating signals with abundant diversities in a “latent-factor-controllable” way.
Those capabilities enable the knowledge transferring through shared causes/factors among different tasks/experiences, emphasized as the important human advantages against the current machine by Lake et al.
and compling with the ideal mental imagery mechanism in memory and thinking.
Beneﬁtted from those capabilities, VAEs have been widely applied to various applications, including disentangled representations learning of images and time series (Higgins et al.
(2016), Kulkarni et al.
(2015), Mathieu et al.
(2016), (Fabius & van Amersfoort (2014)),
few-shot and transfer learning (Rezende et al.
, Higgins et al.
(2017b), Higgins et al.
(2017a)), causal relationships modeling (Louizos et al.
, pixel trajectory predicting (Walker et al.
, joint multi-modal inference learning (Suzuki et al.
increasing diversity in imitation learning (Wang et al.
, generation with memory (Li et al.
(2016)) and etc.
However, the lack of public theoretical study regarding the generating and inference procedure induced by VAEs is tripping the research process:
• Intrinsic Dimension Issue : Could the VAE learn the intrinsic number of factors underlying the data?
• Real Factor Issue : Could the VAE learn the real generating factors underlying the data or just some fantasies?
• Disentanglement Issue : What are the need and range induced by the word “disentanglement”?
• Indicator Issue : Could the effectiveness of current disentanglement metric be guaranteed?
All those lacks distillation would make researchers hard to effectively compute through their knowledge from the experiments and incline to make some avoidable arguments and considerations.
This paper will ﬁrst discuss the properties of the idealistic VAE 1 to target the aforemention issues and then moving to
• Implementation Issue : Could the aforemention analysis be instructive in real implementation?
For Intrinsic Dimension Issue , the information conservation theorem shows that idealistic VAE learns and only learns the intrinsic factor dimension illustrated in Fig.(1).
To Disentanglement Issue , the clariﬁcation on disentanglement is subsequently made.
For Real Factor Issue , the factor equivalence properties shows that idealistic VAE possibly learns all factors in equivalent class rather than exactly pre-speciﬁed real factors.
For Indicator Issue , limitations of the current disentanglement metric are analyzed and several new indicators are introduced.
After that, for Implementation Issue , we relax the discussion to the case that decoding procedure are not deterministic through noise modeling perspective.
The experiments empirically testify that the knowledge derived form the idealistic case could be applied to the realistic sampling case and demonstrate the behaviors of different noise assumptions as well as our indictors.
[Figure]
G AUSSIAN -VAE M ODEL
Gaussian-VAE (Kingma & Welling (2013), Rezende et al.
(2014)) is an scalable unsupervised representation learning model (Higgins et al.
and since Gaussian distribution can be continuously and reversibly mapping to many other distributions, the theoretical analysis on it is also instructive for other continuous latent factors VAE.
Gaussian-VAE assume that input x is generated by several independent Gaussian factors z , that is p θ ( z ) =
N ( z | 0 , I H ) .
The generating/decoding process is modeled as p θ ( x | z ) and the inference/encoding process q φ ( z | x ) is treated as the approximate posterior distribution.
Both of them are parameterized by the neural network with parameter θ and φ .
In VAE setting, the approximate inference method is applied to maximizing the variational lower bound of p θ
[Equation]
[Equation]
2. I DEALISTIC VAE
In order to get assess to aforemention issues, we will start the analysis from the idealistic situation:
An idealistic VAE model means that it can perfectly encode the signal into “used” factors and perfectly decode the “used” factors to original input signal and the factors follows i.i.d unit Gaussian distribution.
The idealistic VAE discussed in this literature should also under the following the Deterministic
Assumption on q φ ( x | z ) and p θ ( z | x ) .
If the factors of x are well understood, the generation process should be deterministic, that is p θ ( x | z )
= δ ( x = ψ ( z )) .
We limit the consideration that q φ ( z | x ) =
δ ( z = ψ − 1 ( x )) is also deterministic as well for simplicity of analysis in this paper.
For more complex situation, this consideration could be also basic and instructive.
We try to address aforemention issues by disregarding the training procedure and direct considering the idealistic VAE’s behavior.
O N I NTRINSIC D IMENSION I SSUE
In order to get asses to the intrinsic dimension issue, we will present the information conservation theorem.
It states some basic truths, e.g. two independent Gaussian and three independent Gaussian cannot be the generating factor of each other under continuous mapping.
The theorem thus further illustrates that idealistic VAE learns and only learns the intrinsic factor dimension.
From the perspective of VAE objective, we will also show that the constraint induced by Gaussian prior, plays the lasso on the mutual information which encourage to clip down the small information dimension and promotes information sparsity in factors.
In order to derive this perspective, the mutual information separation theorem and objective decomposition theorem are subsequently raised.
3. I NFORMATION C ONSERVATION
Theorem 1 (Information Conservation) .
Suppose that z = ( z 1 , · · · , z H ) and y
=
( y 1 , · · · , y P ) are sets of H and P ( H (cid:54)
= P ) independent unit Gaussian random variables, respectively, then these two sets of random variables can not be the generating factor of each other.
That is, there are no continuous functions f : R H
→ R P and g : R P → R H such that
[Equation]
Proof in Appendix B.
The principle of the theorem is visually illustrated in Fig.
3. S EPARATION OF THE M UTUAL I NFORMATION
The mutual information regarding the factors learned by the inference/encoder network and the signal x can be a good quantity for evaluating the generating inﬂuence.
That is,
[Equation]
In order to understand and estimate which factor of the VAE was learnt and inﬂuenced the generating process, I encoder ( x ; z h ) can be taken as a rational indicator 2 .
If we assume that z 1 , z 2 , · · · , z H is conditional independent given x 3 , it can yield a useful result as the following.
Theorem 2 (Mutual Information Separation) .
Let z 1 , · · · , z H be independent unit Gaussian distribution, and z 1 , z 2 , · · · , z H be conditional independent given x .
Then
[Equation]
Proof in Appendix B. This theorem suggests that if the learnt q φ ( z ) can factorize and the q φ ( z | x ) can factorize, then the consideration of each I encoder ( z h ; x ) won’t be excess or lose information.
Besides, when those term comes in the optimization objective, then it can start the negotiation between the information preservation and dimension reduction and play the role of lasso that clip down the factor in dimension with small mutual information.
Theorem 3 (Objective Decomposition) .
The terminology follows the aforemention deﬁnitions and if the involved KL-divergence and mutual information is well deﬁned, then
[Equation]
Proof in Appendix B.
The theorem demonstrates that the second term in variation lower bound in
Eq.
(11) is capable of controlling both the mutual information of x and z induced by the encoder network as well as the similarity of the learnt q φ ( z ) and the prior p θ ( z ) .
Further, the theorem suggests that it possesses the lasso capacity of clipping down the non-intrinsic factor dimension to some extent, visually demonstrated in Fig.(2). [Figure]
I e n c o d e r ( x ; z h )( n a t s )
[Figure]
inﬂuential factor traversals.
The top pulse subgraph: ˜ I encoder ( x ; z h ) of each factor.
The bottom reverse pulse subgraph: the estimated variance σ 2 z h of each factor.
The montages: inﬂuential factor traversals.
We select those factor traversals with visually most interpretable/comprehensive effects to present and the whole inﬂuential factor traversals are listed in appendix 13.
The phenomenon of the multiple semantic change induced by the same learnt factor and the encoding of same semantic among different learnt factor tallies with factor equivalence class theorem 4.
The similar plot of its counterpart with speciﬁed normalized noise can be found in Fig.(16) in Appendix.
O N D ISENTANGLEMENT I SSUE
Inspiration on Disentanglement:
According to the information conservation theorem 1, the independent unit Gaussian factor assumption forms a strong inductive bias and facilitates the model incline to achieve most efﬁcient coding.
Under this assumption, the number of the learnt “used” factors of idealistic VAE should be the same as the true factors number under some assumptions such as the learnt q φ ( z ) should equal p θ ( z ) and decode/encode process is continuous and reversible.
Empirically, sometimes, though the number latent of factors sometimes is pre-speciﬁed larger, only a small amount of unit Gaussian variables regarding the factors of VAE have been used while q φ ( z | x ) is close to deterministic.
The theorem helps provide an interpretation to explain this phenomenon.
Here, in order to avoid the ambiguity of the terminology of disentanglement in this paper, we make the following clariﬁcation.
• The disentanglement of the learnt representation/factors in this literature refers to two parts depicted in Theorem 1: – the factors are closer to be independent with each other, – the factors incline to be able to generate the oracle signal and to be inferred perfectly from the oracle signal through a continuous procedure/mapping.
• The “disentanglement” refers to the closeness of the learnt factors to the pre-speciﬁed independent factors/concpets that can generate the oracle signal and be perfect inferred through a continuous procedure/mapping such as the independent semantic/visual factors.
Therefore, the estimation for D KL ( q φ ( z ) ||
p θ ( z )) that reﬂects the divergence of the learnt factor distribution and
the i.i.d. unit Gaussian prior can be good a indicator to supervise the independence of the factors and served to quantitatively assess the disentanglement of each extracted factor, while the similarity regarding the original signal and reconstruction place another part of the disentanglement.
The “disentanglement” will be shown hard to be obtained in an unsupervised manner.
Concretely, even in the idealistic cases, the extracted factors tend to possess the intrinsic number of latent factors of the model, while there are still possibly large variations of these factors due to it can be obtained only in the equivalent class induced by the pre-speciﬁed factors as proved in the next section.
O N R EAL F ACTOR I SSUE
As for real factor issue, Gaussian Factor Equivalence theorem, (i.e. linear orthogonal transformation of Gaussian factor set are still gaussian factor set.), and Linear Factor Equivalence Class will be presented.
They states that idealistic VAE are possibly learns any factors set in the factors equivalence class, and we should not expect “one-to-one” correspondence by disentanglement. [Figure] VAE trained on CelebA.
The seed image is obtained out from the datasets.
Each block represents the traversal of the generating factor from [ − 3 + z seedh , +3 + z seedh ] .
(a) corresponds to face color white-yellow & female-male change. (b) corresponds to face color white to yellow change. (c) corresponds to background yellow to blue change. (d) corresponds to hair color white to black & face width change.
It can be seen that changing one factor results in multiple semantic factor changes in a comprehensible manner rather than the “one-to-one” correspondence which reﬂexed analysis regarding generating factor equivalence.
5. F ACTORS E QUIVALENCE
Theorem 4 (Gaussian Factor Equivalence) .
Suppose that z = ( z 1 , · · · , z H ) is a set of H independent unit Gaussian random variables.
Let Q ∈ R H
× H be an orthogonal matrix and then y =
Qz is also a set of H independent unit Gaussian random variables.
Besides, z and y can generate each other through a linear homeomorphism mapping. loosen.
If we further demand the enhancement of the pattern separation and completion ability, that is, to make the hyperspheres induced by the observation points in the factor domain fully ﬁll up the whole compact factor ball, then some auxiliary constrains including the restriction on mutual information I encoder ( x ; z ) (deﬁned in Section 3.2) induced by the encoder network need to be introduced.
Proof in Appendix B. This theorem implies that there are a class of unit Gaussian random variables which can generate each other and have equivalent conservation information, as indicated by the following theorem.
Theorem 5 (Linear Gaussian Factor Equivalence Class) .
[Equation]
Then ∀ y ∈ [ z ] , y is a set of H independent unit Gaussian random variables and can generate z through an linear homeomorphism mapping.
The theorem clariﬁes that if Gaussian-VAEs have an linear matrix multiplication freedom degree of learning the factors, then the factors in the equivalence class can all be possibly learnt.
The empirically results tally with the above analysis(see Fig. 3).
Suppose the visual semantic concepts can be viewed as a set of independent Gaussian variables ( z = ( z rotation , z gender , z with − glass , · · · ) T ) which are desired to be captured and learnt by VAEs, while the model is also possible to learn the independent factor set y
= ( y 1 , · · · ) T =
Qz in the equivalence class [ z ] .
This explains why changing one factor like y 1 sometimes empirically results in change in multiple visual concepts.
This perspective suggests that it’s actually hard to obtain the “disentangled representation” that exactly “one-to-one” corresponds to the “independent semantic representation” even though they are in the same equivalence class.
As a result, the idealistic VAE model just tend to learn the “entangle representation” if we do preset “oracle generating factors” belonging to the equivalent class.
However, though those conclusions might be upsetting, it seems not be biology impossible.
A neuron in hippocampus of animals was suggested to combinatorially possess several representation capabilities.
E.g., Aronov et al.
(2017) found that some neurons in rat’s hippocampus involved in spatial representation also were involved in representing sound frequencies after training rats by a tasks that required them to use a joystick to manipulate sound in frequency continuously.
O N I NDICATOR I SSUE 6. L IMITATION OF THE E XISTING D ISENTANGLEMENT M
Higgins et al.
(2016) proposed a “simulated factor” based “disentanglement” metric on the simulation datasets.
However, according to Gaussian factor equivalence theorem 4 that even idealistic
VAE will still learn the factors in the equivalence class, their metric could be effective sometimes for disentanglement but might suffer instability when evaluating the VAE in different trials (detailed in Appendix C).
Further, this metric could be hardly calculated in the real datasets to provide direct feedback of the “disentanglement”.
The reason is that it must pre-know the generating factors expected to be learnt.
6. P ROPOSED INDICATORS
In order to quantify the disentanglement performance 5 as well as the I encoder ( x ; z ) , we assume that q ∗ ( z ) is a factorized zero mean Gaussian estimation for q φ ( z ) .
We can then list the indicators for assessing latent factor disentanglement:
Deﬁnition 1 (Estimation for
[Equation]
it an effective way.
Deﬁnition 2 (Estimation for I encoder ( x ; z ) ) .
[Equation]
Deﬁnition 3 (Estimation for I encoder ( x ; z h ) which quantiﬁes the inﬂuence of each factor) .
[Equation]
Deﬁnition 4 (Estimation for D KL ( q φ ( z [Equation]
| [Equation] p θ
[Equation]
z [Equation] [Equation] Note that the above indicators 2-4 need the value of q ∗ ( z ) , we now introduce how to calculate this term based on Theorem 3.
Through the minimization equivalence, we know that
[Equation]
the q ∗ ( z ) can then be obtained by gradient method from solving the following optimization problem.
[Equation]
I MPLEMENTATION I SSUE 7. VAE WITH N OISE M ODELING
In order to testify the idealistic consideration in real situation, we are not going to learn all the factors or equivalently we assume that datasets have noise 6 (this situation correspond to that p θ ( x | z ) is not deterministic since the major factors z only forms a subset of the whole factors.
7 ), we integrate the noise modeling into our model:
[Equation]
where σ 2 is either manually enumerated or adaptive learned.
Noise modeling can be found crucial in inﬂuencing disentanglement in experiment since it would actually deﬁne the factors aimed to be learnt and subsequently inﬂuence the learnt intrinsic dimension suggested by information conservation property of VAE.
7. N OISE M ODELLING W ITH A UXILIARY C ONSTRAINT
The entangled representation can be caused by the over-large of searching space of q φ ( z | x ) .
If the learned q φ ( z ) = (cid:82) q φ ( z | x ) p data ( x ) dx has a big divergence to p θ ( z ) , then the VAE model tends to learn the entangle representation as it violates the one part of the disentanglement (clariﬁed in section 4).
Actually, in the VAE model, what we want is to search in the space that q φ ( z ) is possibly similar to p θ ( z ) 8 .
By implementing this ideal, we add auxiliary upper bound E x
[Equation]
x
[Equation] q φ
[Equation]
z [Equation] x [Equation]
p
[Equation]
z ))
(detailed in Theorem 3) of D KL ( q φ ( z ) ||
p θ ( z )) to the original objective.
This equivalently leads to the approach of β -VAE raised by Higgins et al.
[Equation]
where β > 1 .
7.2. R ELATION OF VAE AND β -VAE UNDER G AUSSIAN NOISE ASSUMPTION WHEN σ 2 IS PRE - SPECIFIED
Equivalent objective of σ 2 pre-speciﬁed Gaussian noise VAE:
[Equation]
Equivalent objective of σ 2 pre-speciﬁed as σ 2 pre Gaussian noise β -VAE:
[Equation]
where we call βσ 2 pre the normalized variance.
It’s shown that when the σ 2 is pre-speciﬁed, manually tuning it is the same as manually tuning β with a ﬁxed σ 2 pre (for example σ 2 pre = 1 ) under Gaussian noise assumption.
This equivalence saves our time for extra experiment studying the behaviors of this two cases and we call those two case noise speciﬁed β -VAE.
E XPERIMENT 8. D ATASET
MNIST is a database of handwritten digits (Lcun et al.
CelebA (Liu et al. (2015)) is a largescale celebfaces attributes datasets and only its images are used in our experiments.
More details are in Appendix D.
The extensive comparison of Gaussian-noise modeling β -VAE and Gaussiannoise with speciﬁed variance ( β )-VAE is made on MNIST based on our indicators to exploring the disentanglement as well as to testify the theorem and analysis from the idealistic case to the realistic sampling case.
The experiments on CelebA will be auxiliary to further support the generating factor equivalence class theorem 9 .
By setting β as different values, we compare the performance of β -VAE with and without prespeciﬁed noise on MNIST.
We speciﬁcally listed the result of β (=1)-VAE in all cases.
More details can be found in Appendix D.2.
8.1. O N I MPLEMENTATION I SSUE
• Noise modeling/speciﬁcation inﬂuence the disentanglement.
The noise speciﬁcations and modeling signiﬁcantly inﬂuence the model evidence quantitatively and reconstruction qualitatively, as clearly shown in Fig.
The noise speciﬁcations signiﬁcantly inﬂuence the divergence regarding q φ ( z ) and p θ ( z ) and noise speciﬁed and noise learning β -VAE achieve similar disentanglement quantitatively based on the similar indicator behaviours of ˜ D KL ( q φ ( z ) ||
p θ ( z )) in Fig.
(5a) and the number of normal variance factors Fig.
(5b) in regard to the normalized variance.
• Auxiliary constraints can inﬂuence the disentanglement in a different way.
The prominent difference induced by auxiliary constraints would be its stronger suppression on ˜
I encoder ( x ; z ) and the number of inﬂuential factors although changing the noise level also possess this capability indirectly.
It’s somewhat obvious to see and compare value of different indictors under the different β setting by sliding on the green/blue line.
The bigger β , the lower ˜ D KL ( q φ ( z ) ||
p
θ ( z )) and roughly the better reconstruction and hypothesis learnt.
However, it’s more interesting that in regard to the normalized variance, noise learning β -VAE enhances the suppression on ˜ I encoder ( x ; z ) , as depicted in Fig.
(6a) and that is comprehensible since β -VAE is minimizing the auxiliary constraints both I encoder ( x ; z ) + D KL ( q ( z )
|| p ( z ))
based on Theorem 3.
[Figure]
E m p i r i c a l V a r i a t i o n a l L o w [Figure] r B
[Figure]
u n
[Figure]
[Figure]
t i m
[Figure]
t ed D K L ( q ( z ) || p ( z )) ( na t s )
[Figure]
8.1. O N I NTRINSIC D IMENSION I SSUE
• ˜
I encoder ( x ; z h ) effectively determines the “used” factors and VAEs incline to learn the intrinsic factor dimension in realistic sampling case when the disentanglement achieves in some extent.
As shown in Fig.
(2) and (16), the indicator ˜
I encoder ( x ; z h ) determines the “used” factors.
According to those ﬁgures, under the suitable noise assumption, VAEs automatically suppress the auxiliary factors and learn the intrinsic factor dimension as it still capable to have good reconstruct abilities and its D KL ( q φ ( z ) ||
p θ ( z )) closer to zero.
This phenomenon has already suggested by the information conservation theorem 1.
8.1. O N R EAL F ACTOR I SSUE
• Factor equivalence is generally hold and VAEs possibly learn any factor set in the equivalence class.
[Figure]
The reﬂections of the generating equivalence properties 5.1, that is, single factor could result in multiple semantic concepts change and same semantic concept could be encoded in different factors, are again well demonstrated by Fig.
(7), Fig.
(2), Fig.
(16) and Table 1.
[Table]
change.
F UTURE W ORK
From the perspective of representation learning:
• It is interesting that the topology properties of oracle signal are used to obtain the proof for the information conservation theorem.
Other situations including that data owns several connected components can be further considered and would uncover the efﬁcient coding properties of discrete factors.
• When q φ ( z | x ) is far from deterministic, the discussion would be crucial for many other general purposes induced by word disentanglement.
Those study may further extend to the case that data containing different dimension manifolds and to the core VAEs’ pattern separation/completion/generalization capabilities.
A A PPENDIX : N OISE M ODELING D EDUCTION A. G AUSSIAN N OISE D EDUCTION
The VAE objective (i.e., the variational lower bound) can be treated as function of the θ and φ where noise variance parameters σ 2 are contained in θ , [Equation]
[Equation]
Here the
SGVB estimator in (Kingma &
Welling
[Equation]
Note that the noise variance σis also taken as an optimization variable in the model, making the model capable of better adapting noise variation of data in practical cases in a totally automatic way, instead of a manually set manner.
Given multiple data points from a dataset X, we can construct an estimator of the mean marginal likelihood lower bound of the full dataset, based on minibatches
[Equation]
where the minibatch X M = { x m } Mm =1 is a randomly drawn sample set of M datapoints from the full dataset X. Such a lower bound also constitutes an important indicator for model evidence in latter experiment.
We call it the empirical variational lower bound (EVLB) in the following.
Note that ˜ L B ( θ, φ, x m ) (cid:39) L ( θ, φ, x m ) and we can deduce that
[Equation]
The last inequality holds due to D KL ( p data ( x ) || p θ ( x )) ≥ 0 .
The alternative optimization strategy can be readily utilized to design the algorithm for solving the model by iteratively updating the noise parameter and the network ones.
During the optimization process, the objective can be monotonically increasing, and thus the algorithm can be guaranteed to be convergent.
The algorithm is summarized as follows:
Optimization for parameters except σ 2 :
Optimization for
[Equation]
(Close form solution in regard to
[Equation]
Direct gradient method to the transformed variable log simga = log σ ∈ R can be implemented to lift the lower bound ˜
L M as a result to increase the likelihood as well.
A. M O G N OISE D EDUCTION
The noise ε in real situation might be more complex than a simple Gaussian, like that existed in real photographs (Plotz & Roth (2017)).
We thus try to further ameliorate the noise setting as a mixture of Gaussian(MoG) noise.
Such noise modeling strategy has been widely veriﬁed to be effective in applications, like matrix factorization (Meng & Torre (2014)) and robust principal component analysis (Zhao et al.
That is, we assume that
[Equation]
Let c d ∈ { 0 , 1 } K be the latent indicator random one-hot variable, (cid:80)
Kk =1 c dk = 1 , for the MoGnoise component of pixel indexed by d .
Let Π =
[ π 1 , · · · , π K ] and Σ =
[ σ 21 , · · · , σ 2 K ] be the ratio and variance of each component, respectively.
Let Π , Σ be contained in θ .
The conditional joint distribution turns to be
[Equation]
[Equation] The posterior distribution q φ ( z, c | x ) can be factorized as q φ ( z | x ) q ( c | x, z ) , where q φ ( z | x ) will be direct learnt and the alternative of q ( c | x, z ) , q ( c | x, e ) will be set to the last step p θ ( c | x, e ) in regard to EM procedure.
The lower bound of log p θ ( x ) is then reformulated as follows:
[Equation]
Similar to the Gaussian case, the reparamerization trick is implemented,
[Equation]
[Equation] where ˜ z =
En ( x ) + Σ z | x 1 / 2 ( x ) e .
By utilizing the SGVB estimator, we get,
[Equation]
Given an input dataset X, we can then construct an estimator to the mean marginal likelihood lower bound of the full dataset, based on minibatches, as follows:
[Equation]
where z m,l =
En ( x m ) + Σ z | x 1 / 2 ( x m ) e ( l ) and the minibatch X M = { x m } Mi =1 is a randomly drawn sample of M datapoints from the full dataset X.
Then let
[Equation]
where z old =
En old ( x ) + Σ oldz | x 1 / 2 ( x ) e , and we can get
[Equation]
The EM algorithm can be naturally employed to solve the model.
The implementation steps are listed as follows:
Step 1.
Expectation Step .
Set q ( c | x m ,
[Equation]
( l ) )
=
p oldθ old ( c | x m , En old (
[Equation]
oldz |
[Equation]
( x
[Equation]
[Equation]
[Equation]
[Equation]
m , l = 1 , · · · , L .
Calculate the expectation of the latent variable c :
[Equation]
where z m,l : z m,lold =
En old ( x m ) + Σ oldz | x ( x m ) e ( l ) .
The Objective in Maximization Step is obtained as the following,
[Equation]
Step 2.
Maximization Step:
Fix: q ( c | x, e ) determined in the Expectation Step.
M
[Equation]
Update [Π , Σ] and { θ, φ } / [Π , Σ] by alternative optimization strategy.
Update Π , Σ : note here z m,l : z m,lold =
En old ( x m ) + Σ oldz | x ( x m ) e ( l ) , and we can easily get the closed-form updating formula for these parameters:
[Equation]
Update { θ, φ } / [Π , Σ] : gradient methods with respect to { θ, φ } / [Π , Σ] .
Note here z m,l = En ( x m )+ Σ z | x ( x m ) e ( l ) .
The algorithm can then be summarized as follows:
Initialize the coefﬁcient of { θ, φ } / [Π , Σ] and the coefﬁcient of noise ε : Π , Σ
Sample e from N (0 , I H ) to obtain e 1 , · · · , e M [One for each element sample in the mini batch in the next step ( L here is set to 1)].
Sample a mini batch X M from p data ( x ) .
Implement EM algorithms as aforementioned (approximate inference for q ( c, z | x ) ): Expectation: calculate γ dmk .
Maximization: update [Π , Σ] , Update { θ, φ } / [Π , Σ] with gradient methods. 5.
Goto 3: Until Trigger End-Criterion.
B A PPENDIX : P ROOF
Proof.
For theorem 1.
Proof by Contradiction.
Suppose those two function exist, and we will show that they will be inverse mapping of each other and the homeomorphism mapping of R H and R P .
Since R H and R P have different topology structures ( P (cid:54) = H ), the homeomorphism mapping will not exist.
[Equation]
Since both f and g are continuous, there is a homeomorphism mapping between R H and R P and it leads to the contradiction.
Proof.
For theorem 4.
We only need to test the mean and variance of y .
[Equation]
Therefore, y is another set of H independent unit Gaussian random variables.
Since x = Q T y , z
and y can generate each other with an linear homeomorphism mapping.
Proof.
For theorem 2,
[Equation]
Proof.
For theorem
[Equation]
B. A UXILIARY E XPLANATIONS FOR I NDICATORS
Corollary 1.
The terminology follows the aforemention deﬁnitions and if the involved KLdivergence and mutual information be well deﬁned then
[Equation]
The proof of corollary 1 is the same as that of theorem 3.
This corollary suggests that the estimation in deﬁnition 2 provides another upper bound for the capacity of the encoder network.
Empirically, this estimation is a much tighter estimation than using the estimation in deﬁnition 1.
Corollary 2.
The terminology follows the aforemention deﬁnitions and if the involved KLdivergence and mutual information be well deﬁned then
[Equation]
The corollary is an direct result of theorem 3 and corollary 1.
It suggests that the estimation in deﬁnition 4 is a lower bound for D KL ( q φ ( z )
|| p θ ( z )) .
Deﬁnition 5 (Another Estimation for D KL ( q φ ( z ) ||
p θ ( z )) ) .
[Equation]
Empirically, ¯ D KL ( q φ ( z ) ||
p θ ( z )) and ˜ D KL ( q φ ( z ) ||
p θ ( z )) shown the same estimation results on MNIST.
Deﬁnition 6 (Another estimation for I encoder ( x ; z ) ) .
[Equation]
Deﬁnition 7 (Another estimation for I encoder ( x ; z h ) which quantiﬁes the inﬂuence of each factor) .
[Equation]
C A PPENDIX : A NALYSIS ON THE “D ISENTANGLEMENT ” M ETRIC RAISED IN β -VAE (H IGGINS ET AL . (2016))
The terminology inherits those in the β -VAE paper.
The main idea of that “disentanglement” metric is to create a statistic point z diff relevant to the model for each simulated factor respectively and then to use a linear classiﬁer to project the statistic point to the corresponding index of the simulated factor.
If the statistic points induced by the model are easy to be separated then the model is thought to learn “disentangled” representation.
Here, we will argue that even for the idealistic VAE model that follows the disentanglement conditons 4 could still receive bad score under that performance metric in some situations.
Suppose that the true simulated factors v follows N (0 , I H ) .
Then the learnt “used” 10 factor z can be in the equivalence class [ v ] according to theorem 4.
Concretely, there exists an orthogonal transformation Q such that z
=
Qv .
Suppose that the simulated factors with index y of v is ﬁxed.
Suppose v 1 y − fixed and v 2 y − fixed are two random variable representing the samples from the y -ﬁxed v .
Then the factors inferred
In order to calculate the statistic point z diff ( y ) =
E | z 1 − z 2 | , we ﬁrst calculate the mean and variance of ( z 1 − z 2 ) .
[Equation]
Therefore, z diff ( y ) can be obtained through the diagonal value of 2 I − 2 q y q Ty .
That is,
[Equation]
From the above equation, the location of statistic point is unique determined by ( q 2 y 1 , · · · , q 2 yH ) .
When ( q 2 y 1 , · · · , q 2 yH ) is close to the vertex of the unit cubic for each y then all the statistic points turn to be easily separated.
However, from the perspective of the objective, all the orthogonal Q s are with the same potential to be learnt.
It seems not to be with a small probability that statistic points of different indexes take similar location.
For instance, if H = 2 and Q = (cid:32)
[Equation]
√
[Equation]
−
[Equation]
√
[Equation]
(cid:33) then z diff (1) =
z diff (2) cannot be separated while the representation still follows the disentanglement conditions.
Among different trials, the Q might contribute to that “disentanglement” metric but also might not.
That explains why that metric is unstable.
D A PPENDIX : E XPERIMENT D ETAILS
We set L to 1, and minibatch size M to be 100 in all practices.
All the pixels value have been linear normalized in to [0,1].
D. C LARIFICATION ON THE CORRECTION ON RESULTS
We ﬁnd in the last version that the code on calculating q ∗ ( z ) is wrong.
Concretely, the objective of the KL-divergence of two Gaussian is incorrect calculated.
That inﬂuences the estimation of ˜
I encoder ( x ; z ) , ˜ I encoder ( x ; z h ) ,
˜ D KL ( q φ ( z ) || p
θ ( z )) and σ 2 z H but has little inﬂuence on the determination of the “used” factors.
So we redo all the experiments on MNIST and delete the relevant results regarding those wrong-calculated indictors on CelebA and Extended Yale Face B.
The new experiment results solve many our past confusions due to the wrong experiment.
They are
• Why σ 2 z h is strongly correlated with ˜ I
( x ; z h ) ?
• Why σ 2 z h of the used factor is always relative small?
• Why can VAE still learn the “disentangled” representation when the learnt ˜ D KL ( q φ ( z ) ||
p θ ( z )) is such big?
Now we know that they are actually not the cases.
Something better is that we ﬁnd our experiment results are more close to our theoretical analysis: when guaranteeing the reconstruction quality in a tolerance range, the smaller ˜ D KL ( q φ ( z ) ||
p θ ( z ))
the less “used” factors are learnt.
That coincides with information conservation theorem: the independent unit Gaussian of the factors assumption facilitates the most efﬁcient coding.
D. M NIST
We split randomly 7000 datapoints according to ratio [0 . 6 : 0 . 2 : 0 . 2] into training set, validation set (no use), testing set.
All the indicators and q ∗ ( z ) are evaluated/calculated on 10000 datapoints belonging to the testing set.
All the seed images used to infer latent code and to draw the traversal come from the testing set.
In all ﬁgures of latent code traversal each block corresponds to the traversal of a single latent variable while keeping others ﬁxed to either their inferred ( β -VAE, VAE).
Each row represents a different seed image used to infer the latent values in the VAE-based models.
β -VAE and VAE traversal is over the [ − 3 , 3] range.
The assumed variance σ 2 of noise speciﬁed Gaussian of VAE models is enumerated from [0 . 0005 , 0 . 001 : 0 .
The β setting for the noise learning β -VAE is enumerated from [0 . 1 , 0 . 5 , 1 , 2 : 2 : 18] .
D. E XTENDED Y ALE F ACE B (G EORGHIADES ET AL . (2001), L EE ET AL . (2005))
We split randomly 2424 datapoints according to ratio [0 . 8 : 0 . 1 : 0 . 1] into training set, validation set (no use), testing set.
The model is training on the training set.
All the seed images used to infer latent code and to draw the traversal come from the 100 datapoints from the testing set.
In all ﬁgures of latent code traversal each block corresponds to the traversal of a single latent variable while keeping others ﬁxed to either their inferred ( β -VAE, VAE). Each row represents a different seed image used to infer the latent values in the VAE-based models.
β -VAE and VAE traversal is over the [ − 3 , 3] range.
The β setting for the noise learning β -VAE is enumerated from [1 , 40 , 80 , 120 , 160] .
D. C ELEB A
We split randomly roughly 200000 datapoints according to ratio [0 . 8 : 0 . 1 : 0 . 1] into training set, validation set (no use), testing set.
The model is training on the training set.
All the indicators and q ∗ ( z ) are evaluated/calculated on 10000 datapoints selected from testing set.
All the seed images used to infer latent code and to draw the traversal come from the 100 datapoints from the testing set.
In all ﬁgures of latent code traversal each block corresponds to the traversal of a single latent variable while keeping others ﬁxed to either their inferred ( β -VAE, VAE).
Each row represents a different seed image used to infer the latent values in the VAE-based models.
β -VAE and VAE traversal is over the [ − 3 , 3] range.
The β setting for the noise learning β -VAE is enumerated from [1 , 30 , 40] .
D. N ETWORK S TRUCTURE
[Table]
E A PPENDIX : A UXILIARY G ENERATING P ICTURE
Note that only the factors with ˜ I encoder ( x ; z h ) > 0 .
5 are shown.
F A PPENDIX : R ELATED W ORK ON D ISENTANGLEMENT
VAE was proposed by Kingma & Welling (2013) and Rezende et al.
(2014) to implement the efﬁcient learning and inference in directed probabilistic models regarding continuous latent variables with intractable posterior distributions and in scalable datasets.
They introduced a network inference/recoginition model to represent the approximate posterior distribution and utilized reparameterization trick for stochastic joint optimization of a variational lower bound containing the parameters of both the generative/decoder and inference/recoginition/encoder models.
After being raised, many VAE variations have been proposed to boost VAE’s capabilities in generation quality and/or disentanglement of the learned representation.
In these methods, multiple efforts were made by improving the generative and inference network structures.
Typical works along this line include the convolution/de-convolution structure raised by Kulkarni et al.
and ladder structure raised by Zhao et al.
Some other works advanced the mechanism under the VAE generation/inference processes.
Typical works include the iterative attention generation/inference mechanism raised by Gregor et al.
, normalizing ﬂow proposed by Rezende & Mohamed (2015) that enhanced the expressive ability of the approximate posterior and its variants (Kingma et al.
Despite the improvement to the VAE itself, some other efforts were made by the ensemble between
GAN with VAE.
E.g., Larsen et al.
(2015) uniﬁed GAN and VAE to obtain a better reconstruction and a high-level abstracts visual features embedding.
Mathieu et al.
(2016) also uniﬁed GAN and
VAE but put emphasis on disentangling factors of variation.
GANs without auxiliary design would learn the data distribution disregarding its noise level though suffer from unstable training and mode collapsing (Salimans et al.
(2016)) while VAEs would assume a decomposition of the noise and oracle clean datapoint regarding the noise data with an auxiliary prior on the distribution regarding the factors.
Besides, many efforts were made by regularization on the factor distribution or factor generating effect.
E.g., Makhzani et al.
(2015) introduced an adversarial loss into the latent space of the autoencoder which in idealistic case could learn any kind factor/lantent distribution including those contributing to the disentangled factors/representation.
InfoGAN, raised by Chen et al.
(2016), introduced the infomax principle to GAN by adding an auxiliary mutual information regularization which enabled the inference of GANs and led to a better disentangled representation as well.
Recently, there is a new VAE variation is proposed by Higgins et al.
(2016) who introduced the β -VAE framework which enhanced the constraints regarding the KL-divergence of the posterior and prior distribution of VAE and showd a novel disentanglement performance.
This method has obtained a better performance as compared with conventional VAE methods, especially on its ﬂexible tuning a compromising a parameter beta between the KL-divergence term and the likelihood term (the variational lower bound).
[Figure]
factor factor
[Figure]
factor
[Figure]
factor factor
[Figure]
Noise Learning -VAE[Vary ]
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
n
[Figure]