P RELIMINARY T HEORETICAL E XPLORATION IN V ARIATIONAL A UTOENCODER AND F ITTING D ATA N OISE IN IT
Anonymous authors
Paper under double-blind review
A BSTRACT
Why does variational autoencoder(VAE) suffer from bad reconstruction, and what inﬂuence the disentanglement of VAE?
This paper tries to preliminarily address those issues theoretically in the idealistic situation and practically through noise modeling perspective in the realistic case.
On one fold, information conservation (e.g. 2 Gaussian and 3 Gaussian cannot be the generating factors of each other), factor equivalence (e.g. the orthogonal transformation of Gaussian factors is still Gaussian factors) and separation of mutual information properties in the idealistic VAE case are revealed. They preliminary but fundamentally answer what would be learned by idealistic VAE and what would not.
Several performance indicators regarding the disentanglement and generating inﬂuence are subsequently raised to evaluate the performance of VAE model and to supervise the used factors.
On other fold, the paper proposes the adaptive noise learning algorithms of Gaussian noise and mixture Gaussian noise assumption which empirically contributes to a better reconstruction than original VAE noise assumptions.
The implementations under noise assumptions empirically testify the theoretical analysis and also show their own characteristic in pursue disentanglement.
I NTRODUCTION
Variational AutoEncoder(VAE)s (Kingma & Welling (2013), Rezende et al.
(2014)) have shown their powerful human-like abilities: modelling causal relationship, unsupervisedly extracting disentangled factors/representation (Bengio et al.
and generating signals with abundant diversities in a “latent-factor-controllable” way.
Those capabilities enable the knowledge transferring through shared causes/factors among different tasks/experiences, emphasized as the important human advantages against the current machine by Lake et al.
and compling with the ideal mental imagery mechanism in memory and thinking.
Beneﬁtted from those capabilities, VAEs have been widely applied to various applications, including disentangled representations learning of images and time series (
However, the lack of public theoretical study regarding the generating and inference procedure induced by VAEs is tripping the research process: we wouldn’t exactly know whether the VAE could learn the real factors or just some fantasies that model itself makes up; we couldn’t exactly clarify the need and range induced by the word “disentanglement”; we actually couldn’t exactly guarantee the effectiveness of the current disentanglement metric.
All those lacks distillation would make researchers hard to effectively compute through their knowledge from the experiments and incline to make some avoidable arguments and considerations.
Further, even though some of the aforemention issues would be addressed in some idealistic situation, the real situation would be further complex and the side effect of ignorance of noise considera- [Figure] tion in VAEs would be prominent.
In VAE assumption, on the one hand, the major factors are learnt and inferred and the noise enables the optimization; on the other hand, more importantly, the prior for major factors and noise modeling together form the core inductive bias of VAE.
In real implementation, different datasets may have drastically different noise and improper modeling for noise inclines to lead to incorrect model hypothesis and inﬂuence learning consequence.
Therefore, the noise speciﬁcation/modeling could signiﬁcantly inﬂuence reconstruction and representation learning ability.
Besides, the applications of VAE has suffered from the blurred reconstructions/generations and the unstable disentangled quality (Higgins et al.
In particular, numbers of applications assume a ﬁxed noise that disables the model to ﬂexible adapt to the real noise (Makhzani et al.
(2015), Walker et al.
, Higgins et al.
(2017a), Doersch (2016)).
These issues, as results, sometimes could make VAE have to be an auxiliary part to only alleviate the generation shortcomings of generative adversarial network(GAN)
(Larsen et al.
(2015), Wang et al.
[Figure]
[Equation]
[Figure]
This paper will ﬁrst discuss the properties of the idealistic VAE, that is, roughly the VAE whose decoding procedure and encoding procedure are both deterministic.
The information conservation theorem, the clariﬁcation on disentanglement, factor equivalence class, separation of mutual information property and several indicators will be introduced.
After that we relax the discussion to the case that decoding procedure are not deterministic.
We introduce Gaussian, Mixture of Gaussian noise modeling regime and discuss network parameterized Gaussian for ﬁtting noise to data.
The experiments are design to testify whether the knowledge derived form the idealistic case would be applied to the realistic sampling case and to study the behavior of different noise assumptions as well as our indictors.
G AUSSIAN -VAE M ODEL
Gaussian-VAE (Kingma & Welling (2013), Rezende et al.
(2014)) is an scalable unsupervised representation learning model (Higgins et al.
and since Gaussian distribution can be continuously and reversibly mapping to many other distributions, the theoretical analysis on it is also instructive for other continuous latent factors VAE.
Gaussian-VAE assume that input x is generated by several independent Gaussian factors z , that is p θ ( z ) =
N ( z | 0 , I H ) .
The generating/decoding process is modeled as p θ ( x | z ) and the inference/encoding process q φ ( z | x ) is treated as the approximate posterior distribution.
Both of them are parameterized by the neural network with parameter θ and φ .
In VAE setting, the approximate inference 1 method is applied to maximizing the variational lower bound of p θ ( x ) =
(cid:82) p θ ( x | z ) p
θ ( z ) dz ,
[Equation]
G ENERATING F ACTOR P ROPERTIES AND P ERFORMANCE I NDICTORS
In order to get assess to the question, whether the VAE could learn the real factors or just some fantasies that model itself makes up, we start the analysis from the idealistic situation 2 .
3. D ETERMINISTIC A SSUMPTION ON q φ ( x | z ) AND p θ ( z | x )
If the factors of x are well understood, the generation process should be deterministic, that is p θ ( x | z )
= δ ( x = ψ ( z )) .
We limit the consideration that q φ ( z | x ) =
δ ( z = ψ − 1 ( x )) is also deterministic as well for simplicity of analysis in this paper.
For more complex situation, this consideration could be basic and instructive.
3. I NFORMATION C ONSERVATION
We try to address this issue by disregarding the training procedure and direct considering the idealistic VAE’s behavior through the following theorem.
Theorem 1 (Information Conservation) .
Suppose that z = ( z 1 , · · · , z H ) and y
=
( y 1 , · · · , y P ) are sets of H and P ( H (cid:54)
= P ) independent unit Gaussian random variables, respectively, then these two sets of random variables can not be the generating factor of each other.
That is, there are no continuous functions f : R H
→ R P and g : R P → R H such that
[Equation]
Proof in Appendix 8.3.
The principle of the theorem is visually illustrated in Fig.
[Equation]
3.2. E FFICIENT R EPRESENTATION AND C LARIFICATION ON D ISENTANGLEMENT
According to the information conservation theorem, the independent unit Gaussian factor assumption forms a strong inductive bias and facilitates the model incline to achieve most efﬁcient coding.
Under this assumption, the number of the learnt “used” factors of idealistic VAE should be the same as the true factors number under some assumptions such as the learnt q φ ( z ) should equal p θ ( z ) and decode/encode process is continuous and reversible.
Empirically, sometimes only a small amount of unit Gaussian variables regarding the factors of VAE have been used and the q φ ( z | x ) is close to deterministic though the number latent of factors sometimes is pre-speciﬁed larger.
This theorem helps provide an interpretation to explain this phenomenon.
Here, in order to avoid the ambiguity of the terminology of disentanglement in this paper, based on the information conservation theorem, we make the following clariﬁcation.
• The disentanglement of the learnt representation/factors in this literature refers to two parts depicted in Theorem 1: – the factors are closer to be independent with each other, – the factors incline to be able to generate the oracle signal and to be inferred perfectly from the oracle signal through a continuous procedure/mapping.
• The “disentanglement” refers to the closeness of the learnt factors to the pre-speciﬁed independent factors/concpets that can generate the oracle signal and be perfect inferred through a continuous procedure/mapping such as the independent semantic/visual factors.
Therefore, the estimation for D KL ( q φ ( z ) ||
p θ ( z )) that reﬂects the divergence of the learnt factor distribution and
the i.i.d. unit Gaussian prior can be good a indicator to supervise the independence of the factors and served to quantitatively assess the disentanglement of each extracted factor, while the similarity regarding the original signal and reconstruction place another part of the disentanglement.
The “disentanglement” will be shown hard to be obtained in an unsupervised manner.
Concretely, even in the idealistic cases, the extracted factors tend to possess the intrinsic number of latent factors of the model, while there are still possibly large variations of these factors due to it can be obtained only in the equivalent class induced by the pre-speciﬁed factors as proved in the next subsection.
3. F ACTORS E QUIVALENCE : W E SHOULD NOT EXPECT “ ONE - TO - ONE ” CORRESPONDENCE BY DISENTANGLEMENT .
Theorem 2 (Gaussian Factor Equivalence) .
Suppose that z = ( z 1 , · · · , z H ) is a set of H independent unit Gaussian random variables.
Let Q ∈ R H
× H be an orthogonal matrix and then y =
Qz is also a set of H independent unit Gaussian random variables.
Besides, z and y can generate each other through a linear homeomorphism mapping.
Proof in Appendix 8.3.
This theorem implies that there are a class of unit Gaussian random variables which can generate each other and have equivalent conservation information, as indicated by the following theorem.
Theorem 3 (Linear Gaussian Factor Equivalence Class) .
[Equation]
Then ∀ y ∈ [ z ] , y is a set of H independent unit Gaussian random variables and can generate z through an linear homeomorphism mapping.
The theorem clariﬁes that if Gaussian-VAEs have an linear matrix multiplication freedom degree of learning the factors, then the factors in the equivalence class can all be possibly learnt. [Figure]
The empirically results tally with the above analysis(see Fig. 3).
Suppose the visual semantic concepts can be viewed as a set of independent Gaussian variables ( z = ( z rotation , z gender , z with − glass , · · · ) T ) which are desired to be captured and learnt by VAEs, while the model is also possible to learn the independent factor set y
=
( y 1 , · · · ) T =
Qz in the equivalence class [ z ] .
This explains why changing one factor like y 1 sometimes empirically results in change in multiple visual concepts.
This perspective suggests that it’s actually hard to obtain the “disentangled representation” that exactly “one-to-one” corresponds to the “independent semantic representation” even though they are in the same equivalence class.
As a result, the idealistic VAE model just tend to learn the “entangle representation” if we do preset “oracle generating factors” belonging to the equivalent class.
However, though those conclusions might be upsetting, it seems not be biology impossible.
A neuron in hippocampus of animals was suggested to combinatorially possess several representation capabilities.
E.g., Aronov et al.
(2017) found that some neurons in rat’s hippocampus involved in spatial representation also were involved in representing sound frequencies after training rats by a tasks that required them to use a joystick to manipulate sound in frequency continuously.
3.3. D EFICIENCY OF THE E XISTING D ISENTANGLEMENT M ETIC
Higgins et al.
(2016) proposed a “simulated factor” based “disentanglement” metric on the simulation datasets.
However, this metric could be hardly calculated in the real datasets and provide direct feedback of the “disentanglement”.
The reason is that it must pre-know the generating factors expected to be learnt.
According to Gaussian factor equivalence theorem 2 that even idealistic VAE will still learn the factors in the equivalence class, their metric could be effective sometimes for disentanglement but might suffer instability when evaluating the VAE in different trials (detailed in Appendix 8.4).
3. S EPARATION OF THE M UTUAL I NFORMATION
The mutual information 4 regarding the factors learned by the inference/encoder network and the signal x can be a good quantity for evaluating the generating inﬂuence.
That is,
[Equation]
In order to understand and estimate which factor of the VAE was learnt and inﬂuenced the generating process, I encoder ( x ; z h ) can be taken as a rational indicator 5 .
If we assume that z 1 , z 2 , · · · , z H is conditional independent given x 6 , it can yield a useful result as the following.
Theorem 4 (Separation of the Mutual Information) .
Let z 1 , · · · , z H be independent unit Gaussian distribution, and z 1 , z 2 , · · · , z H be conditional independent given x .
Then
[Equation]
Proof in Appendix 8.3.
This theorem suggests that if the learnt q φ ( z ) can factorize and the q φ ( z | x ) can factorize, then the consideration of each I encoder ( z h ; x ) won’t be excess or lose information.
Besides, when those term comes in the optimization objective, then it can the negotiation between the information preservation and dimension reduction.
While if the separation of mutual information doesn’t hold, the negotiation could suffer from the exponential selection complexity in dimension.
3. I NDICATORS
In order to quantify the disentanglement performance 7 as well as the I encoder ( x ; z ) , we assume that q ∗ ( z ) is a factorized zero mean Gaussian estimation for q φ ( z ) .
We ﬁrst propose the following relevant theorem and then provide the indicators.
Theorem 5.
The terminology follows the aforemention deﬁnitions and if the involved KL-divergence and mutual information is well deﬁned, then
[Equation]
Proof in Appendix 8.3.
The theorem demonstrates that the second term in variation lower bound in
Eq.
(14) is capable of controlling both the mutual information of x and z induced by the encoder network as well as the similarity of the learnt q φ ( z ) and the prior p θ ( z ) (one part of disentanglement performance).
We can then list the indicators for assessing latent factor disentanglement:
Deﬁnition 1 (Estimation for E x ∼ p data ( x ) D KL ( q φ ( z | x ) || p
θ ( z )) )
[Equation]
Deﬁnition 2 (Estimation for I encoder ( x ; z ) ) .
[Equation]
Deﬁnition 3 (Estimation for I encoder ( x ; z h ) which quantiﬁes the inﬂuence of each factor) .
[Equation]
Deﬁnition 4 (Estimation for D KL ( q φ ( z ) || p
θ ( z )) ) .
[Equation]
that the most of the visual concept/factors follow the assumption regarding the “disentanglement”, it is rational to qualitatively measure the interpretability of extracted latent factors by human perception to infer the disentanglement.
Besides, previous empirical evidences of VAE applications (Higgins et al.
, Higgins et al.
(2017b), Larsen et al.
(2015), Mathieu et al.
(2016)) suggest it an effective way.
Note that the above indicators 2-4 need the value of q ∗ ( z ) , we now introduce how to calculate this term based on Theorem 5.
Through the minimization equivalence, we know that
[Equation]
the q ∗ ( z ) can then be obtained from solving the following optimization problem which can be calculated by gradient method.
[Equation]
VAE WITH N OISE M ODELING
When the consideration meets the real situation, the numbers of the factors underlaying x can be tremendous.
Its not necessary and even possible to learn them all, especially for those less contribute to the generating as well as the latter potential tasks.
Let we roughly call those factor as noise.
More concretely, this situation correspond to that p θ ( x | z ) is not deterministic since the major factors z forms a subset of the whole factors.
Since VAEs learn in a purely unsupervised manner, the speciﬁcation of the noise distribution p θ ( x | z ) would limit the freedom of the learned major factors and deﬁne the major factor parts to some extent.
The superiority brought by proper consideration on noise modeling has been veriﬁed in various applications like denoising (Chen et al.
background substraction (Yong et al.
(2017)), derain (Wei et al.
medical image reconstruction (Xie et al.
(2017)), detecting differentially expressed genes (Lopez et al.
(2017)) and ect.
In this section, we take the Gaussian variance parameter as an optimization variable and integrate it into the VAE model to make the noise ﬁtted by data, and furthermore, we ameliorate the noise as mixture of Gaussian 9 to further enhance its noise modeling capability.
The β -VAE is employed to integrate such noise modeling mechanism and also place as the comparison to the noise speciﬁcation/modelling.
[Figure]
4. G AUSSIAN C ASE
When modelling the real-valued generation process, VAEs often assume the conditional distribution to be
[Equation]
G is the generating/decoder function parameterized by neural networks and σ 2 is the variance parameter of the Gaussian distribution.
The data are formulated by the ideal generation G ( z ) (oracle part of data) corrupted by an additional element-wise Gaussian noise ε with variance σ 2 .
10 That is,
[Equation]
G
[Equation]
Since the objective of the log-likelihood needs to negotiate with the KL-divergence term which controls one perspective of the disentanglement depicted in Theorem 5 in the optimization process for VAE, if the noise setting cannot guarantee to comply with the goal and prior, which naturally tends to lead to more unpredictable performance of the method in practice.
We can easily integrate the noise parameters into the VAE objective to make the noise as a learnable part in VAE to compensate a better performance on both representation and reconstruction of VAE and if necessary, more prior knowledge of noise could also be introduced to the objective.
4. M IXTURE OF G AUSSIAN C ASE
The noise ε in Eq.
(11) in real situation might be more complex than a simple Gaussian, like that existed in real photographs (Plotz & Roth (2017)).
We thus try to further ameliorate the noise setting as a mixture of Gaussian(MoG) noise.
Such noise modeling strategy has been widely veriﬁed to be effective in applications, like matrix factorization (Meng & Torre (2014)) and robust principal component analysis (Zhao et al.
That is, we assume that
[Equation]
In order to optimize under this noise assumption, we implement empirical EM-like SGVB algorithm detailed in appendix 8.2.
4. N ETWORK P ARAMETERIZED G AUSSIAN C ASE
Here we want to give a short discussion on the difference of the proposed VAE model with noise modeling with that raised by Kingma & Welling (2013), in which they model the noise as the following parameterized network structure: [Equation]
[Equation]
where h represents the mapping induced by the previous network layers; σ 2 ( z ) represents the diagonal of diagonal covariance matrix; { W u , W σ , b u , b σ } are the weights and biases of the last layer of network.
This noise assumption has very strong expressibility.
The model assumes that each pixel noise indexed by d has its own level determined by b σd and is also inﬂuenced by the deterministic part
W σ h ( z ) .
When the noise level is shared among all data points and not inﬂuenced by the deterministic part, then it degenerates to the Gaussian assumption; if the noise has several discrete level, it then tends to degenerate to MoG assumption on noise.
However, on the one hand, the assumption used in the model inclines to make the optimization for the model difﬁcult due to the numerical instability caused even by one zero-variance residual pixel.
This is possibly why most applications on VAE have not employed this assumption while prefer to ﬁx and manually set a noise level before VAE training.
On the other hand, the over-parameterized noise could signiﬁcantly increase the difﬁculty to ﬁnd a better deterministic part since the model might be inclined to ﬁt the noise hypothesis rather than learn a good G .
These limitations have been empirically veriﬁed by all our experiments and can be observed in the experimental results as listed in Section 5. [Equation] Origin Gaussian
MoG
[Figure] Network [Figure] 4. N OISE M ODELLING W ITH A UXILIARY C ONSTRAINT
The entangled representation could be caused by the over-large of searching space of q φ ( z | x ) .
If the learned q φ ( z ) = (cid:82) q φ ( z | x ) p data ( x ) dx has a big divergence to p θ ( z ) , then the VAE model tends to learn the entangle representation as it violates the one part of the disentanglement (clariﬁed in section 3.2.1).
Actually, in the VAE model, what we want is to search in the space that q φ ( z ) is possibly similar to p θ ( z ) 11 .
By implementing this ideal, we add auxiliary upper bound E x [Equation] x ) D KL ( q φ ( z | x ) || p θ ( z ))
(detailed in Theorem 5) of D KL ( q φ ( z ) ||
p θ ( z )) to the original objective.
This equivalently leads to the approach of β -VAE raised by Higgins et al.
[Equation]
where β > 1 .
E XPERIMENT 5. Q UESTIONS
The experiments will try to give the answer to the following question:
• Would noise modeling/speciﬁcation inﬂuence the disentanglement?
• What’s the difference characteristic of disentanglement induced by auxiliary constraints?
• How do different noise assumptions including the Gaussian, Mixture of Gaussian and Network Parameterized Gaussian act?
• Would the information conservation theorem be instructive in real implementation?
• Would the generating factor equivalence class properties exist in real implementation?
• How do our indictors act and whether they can determine the used the factors or not?
5. D ATASET
MNIST is a database of handwritten digits (Lcun et al.
CelebA (Liu et al. (2015)) is a large-scale celebfaces attributes datasets and only its images are used in our experiments.
Extended
Yale Face B contains (Georghiades et al.
Lee et al.
images of several human subjects under different poses and different illumination conditions.
More details are in Appendix 8.5.
The extensive comparison of Gaussian-noise modeling β -VAE and Gaussian-noise with speciﬁed variance ( β )-VAE is made on MNIST based on our indicators to exploring the disentanglement as well as to testify the theorem and analysis from the idealistic case to the realistic sampling case.
The experiments on Extended Yale Face B will focus on the comparison of behavior regarding
Gaussian, Mixture of Gaussian and Network Parameterized Gaussian.
The experiments on CelebA will be auxiliary to further support the generating equivalence class theorem.
5. E XPERIMENTS ON MNIST 5.3. R ELATION OF VAE AND β -VAE UNDER G AUSSIAN NOISE ASSUMPTION WHEN σ 2 IS PRE - SPECIFIED
Equivalent objective of σ 2 pre-speciﬁed Gaussian noise VAE:
[Equation]
Equivalent objective of σ 2 pre-speciﬁed as σ 2 pre Gaussian noise β -VAE:
[Equation]
where we call βσ 2 pre the normalized variance.
It’s shown that when the σ 2 is pre-speciﬁed, manually tuning it is the same as manually tuning β with a ﬁxed σ 2 pre (for example σ 2 pre = 1 ) under Gaussian noise assumption.
This equivalence saves our time for extra experiment studying the behaviors of this two cases and we call those two case noise speciﬁed β -VAE.
5.3. N OISE MODELING / SPECIFICATION INFLUENCE THE DISENTANGLEMENT
By setting β as different values, we compare the performance of β -VAE with and without prespeciﬁed noise on this dataset.
We speciﬁcally listed the result of β (=1)-VAE in all cases.
More details can be found in Appendix 8.5.2.
The noise speciﬁcations and modeling signiﬁcantly inﬂuence the model evidence quantitatively and reconstruction qualitatively, as clearly shown in Fig.
The noise speciﬁcations signiﬁcantly inﬂuence the divergence regarding q φ ( z ) and p θ ( z ) and noise speciﬁed and noise learning β -VAE achieve similar disentanglement quantitatively based on the similar indicator behaviours of ˜ D KL ( q φ ( z ) ||
p θ ( z )) in Fig.
(7a) and the number of normal variance factors Fig.
(7b) in regard to the normalized variance.
5.3. D IFFERENCE INDUCED BY AUXILIARY CONSTRAINTS
The prominent difference induced by auxiliary constraints would be its stronger suppression on ˜
I encoder ( x ; z ) and the number of inﬂuential factors although changing the noise level also possess [Figure] E m p i r i c a l V a r i a t i o n a l L o w
[Figure]
r B [Figure] u n [Figure]
[Figure]
t
[Figure]
t
[Figure]
D K L ( q ( z ) || p ( z )) ( na t s )
[Figure]
this capability indirectly.
It’s somewhat obvious to see and compare value of different indictors under the different β setting by sliding on the green/blue line.
The bigger β , the lower ˜ D KL ( q φ ( z ) ||
p
θ ( z )) and roughly the better reconstruction and hypothesis learnt.
However, it’s more interesting that in regard to the normalized variance, noise learning β -VAE enhances the suppression on ˜ I encoder ( x ; z ) , as depicted in Fig.
(8a)
and that is comprehensible since β -VAE is minimizing the auxiliary constraints both I encoder ( x ; z ) + D KL ( q ( z )
|| p ( z ))
based on Theorem 5.
5.3. ˜ I encoder ( x ; z h ) EFFECTIVELY DETERMINES THE “ USED ” FACTORS AND INFORMATION CONSERVATION HOLD WHEN THE DISENTANGLEMENT ACHIEVES IN SOME EXTENT .
As shown in Fig.
(9) and (10), the indicator ˜
I encoder ( x ; z h ) determines the “used” factors.
According to those ﬁgures, VAEs automatically suppress the auxiliary factors and learn the factor dimension as it still capable to have good reconstruct abilities and its D KL ( q φ ( z ) ||
p θ ( z )) closer to zero under the suitable noise assumption.
This phenomenon has already suggested by the information conservation theorem 1.
[Figure]
[Figure]
In all ﬁgures of factor traversal each montage corresponds to the traversal of a single factor while keeping others ﬁxed to their inferred.
Each row corresponds to a different seed image used to infer latent factor value in the VAE-based models.
The traversal is over [-3, 3].
Note that all the factors with ˜ I encoder ( x ; z h ) not close to zero ( > 0 . 1) can be visually told the existence of their generating effect.
We select those factor traversals with visually most interpretable/comprehensive effects to present and the whole inﬂuential factor traversals are listed in appendix 17.
The mutual information of “used” factor learnt by noise learning β -VAE can be found relatively balanced.
The phenomenon of the multiple semantic change induced by the same learnt factor and the encoding of same semantic among different learnt factor tallies with factor equivalence class theorem 2.
5. E XPERIMENTS ON E XTENDED Y ALE F ACE D ATABASE B
In this series, we compare the β -VAEs with noise Gaussian, MoG, and network parameterized Gaussian noise components.
The Table 1 quantitatively compare the EVLB performance indicator of different methods.
5.4. M O G IS MORE DEDICATED .
β -VAE with MoG noise modeling ( β = 1 ) learns an evidently better distribution hypothesis compared with the Gaussian one.
It’s comprehensive that MoG-VAE learns two different noise level component according to Fig.
One of them could be interpreted as the intrinsic physical Gaussian noise and the other might be the part which is hard to be reconstructed.
However, if the noise is assumed to be one Gaussian, then it can hardly decompose such an elaborate description of noise conﬁgurations, as shown in Fig.
I e n c o d e r ( x ; z h )( n a t s )
[Figure]
2 z h
Thickness
ThicknessRound
[Figure]
[Table]
The result can be inﬂuenced by the speciﬁcation of the initialization of the σ 2 k and π k .
Variance was clipped to 0.0001 to guarantee the stable optimization.
The results of Network(MoG-2) and
Network(G) are derived using the ﬁxed G ( z ) but changing the noise hypothesis and recalculating the noise parameters.
5.4. N ETWORK P ARAMETERIZED G AUSSIAN INCLINES TO OVERFIT .
For the parameterized VAE, although the network parameterized noise VAE achieves the highest distribution hypothesis, however the qualitatively reconstructions of the network are not as good as its EVLB.
The model generates more blurred reconstructions, which can be observed on two typical faces shown in Fig.
[Table] We further plug the generator G ( z ) into MoG-2 noise hypothesis, according to the Table 1, its
EVLB decreases signiﬁcantly.
Besides, the network parameterized noise VAE suffers severely from the numerical instability such that we could not ﬁnish a complete training(2002 epoch) due to the its objective collapsed illustrated in Section 4.3.
All of the aforemention evidences suggest that the model learns a relatively dedicate hypothesis for the noise rather than the deterministic part(oracle signal).
5. E XPERIMENTS ON C ELEB A
The reﬂection of the generating equivalence properties 3.3, that is, single factor could result in multiple semantic concepts change and same semantic concept could be encoded in different factors, are again well demonstrated by Fig.
(11) and Table 2.
P ERSPECTIVE
We further try to give some discussions which should be beneﬁcial to our future works on this work.
Firstly, from the perspective of noise modeling:
[Table] • The physical noise in different practical scenarios, such as medical image processing/generating, can be taken into consideration while implementing the VAEs.
• The noise modeling for other generative model and deep model is also an interesting direction.
From the perspective of representation learning:
• It is interesting that the topology properties of oracle signal are used to obtain the proof for the information conservation theorem.
Other situations including that data owns several connected components can be further considered and would uncover the efﬁcient coding properties of discrete factors.
• When q φ ( z | x ) is far from deterministic, the discussion would be crucial for many general purposes induced by word disentanglement.
Those study may further extend to the case that data containing different dimension manifolds and to the core VAEs’ pattern separation/completion/generalization capabilities.
R ELATED W ORK
VAE was proposed by Kingma & Welling (2013) and Rezende et al.
(2014) to implement the efﬁcient learning and inference in directed probabilistic models regarding continuous latent variables with intractable posterior distributions and in scalable datasets.
They introduced a network inference/recoginition model to represent the approximate posterior distribution and utilized reparameterization trick for stochastic joint optimization of a variational lower bound containing the parameters of both the generative/decoder and inference/recoginition/encoder models.
While they also designed a network for parameterizing the noise for Gaussian MLP decoder, the capability of noise rectifying based on data has not been speciﬁcally emphasized, just similar to most of the latter VAE applications, which easily specify a ﬁxed Gaussian noise (with preset variance parameter) but more or less underestimate the role of noise learning.
After being raised, many VAE variations have been proposed to boost VAE’s capabilities in generation quality and/or disentanglement of the learned representation.
In these methods, multiple efforts were made by improving the generative and inference network structures.
Typical works along this line include the convolution/de-convolution structure raised by Kulkarni et al.
and ladder structure raised by Zhao et al.
Some other works advanced the mechanism under the VAE generation/inference processes.
Typical works include the iterative attention generation/inference mechanism raised by Gregor et al.
, normalizing ﬂow proposed by Rezende & Mohamed (2015) that enhanced the expressive ability of the approximate posterior and its variants (Kingma et al.
Despite the improvement to the VAE itself, some other efforts were made by the ensemble between
GAN with VAE.
E.g., Larsen et al.
(2015) uniﬁed GAN and VAE to obtain a better reconstruction and a high-level abstracts visual features embedding.
Mathieu et al.
(2016) also uniﬁed GAN and
VAE but put emphasis on disentangling factors of variation.
GANs without auxiliary design would learn the data distribution disregarding its noise level though suffer from unstable training and mode collapsing (Salimans et al.
(2016)) while VAEs would assume a decomposition of the noise and oracle clean datapoint regarding the noise data with an auxiliary prior on the distribution regarding the factors.
If the wrong speciﬁed noise in VAE is combined into GAN which inclines to conduce unexpected images with corruptions, it’s reasonable that the ensemble model may achieve a better hypothesis of true data distribution.
Besides, many efforts were made by regularization on the factor distribution or factor generating effect.
E.g., Makhzani et al.
(2015) introduced an adversarial loss into the latent space of the autoencoder which in idealistic case could learn any kind factor/lantent distribution including those contributing to the disentangled factors/representation.
InfoGAN, raised by Chen et al.
(2016), introduced the infomax principle to GAN by adding an auxiliary mutual information regularization which enabled the inference of GANs and led to a better disentangled representation as well.
Recently, there is a new VAE variation is proposed by Higgins et al.
(2016) who introduced the β -VAE framework which enhanced the constraints regarding the KL-divergence of the posterior and prior distribution of VAE and showd a novel disentanglement performance.
This method has obtained a better performance as compared with conventional VAE methods, especially on its ﬂexible tuning a compromising a parameter beta between the KL-divergence term and the likelihood term (the variational lower bound).
We thus select this VAE model to embed our noise modeling regime to improve its capability of adapting real noise in data.
In the other perspective of the decomposition of the noise and oracle clean data, deep denoising models, including denoising autoencoder (Goodfellow et al.
(2016)), stacked denoising autoencoders (Vincent et al.
denosing variational encoder (Im et al.
etc., have also shown similar ability to the proposed VAE model with noise modeling.
However, those models are trained by using paired/supervised data (both corrupted and clean data) while our model just learn in a purely unsupervised manner without supervision regarding the noise/or oracle data.
Just similar to traditional
VAE, such implementation paradigm is more similar to human learning process, and can be better generalized into real applications lacking supervision knowledge.
A PPENDIX 8. G AUSSIAN N OISE D EDUCTION
The VAE objective (i.e., the variational lower bound) can be treated as function of the θ and φ where noise variance parameters σ 2 are contained in θ ,
[Equation]
Here the
SGVB estimator in (Kingma &
Welling
[Equation]
[Equation]
used.
Note that the noise variance σis also taken as an optimization variable in the model, making the model capable of better adapting noise variation of data in practical cases in a totally automatic way, instead of a manually set manner.
Given multiple data points from a dataset X, we can construct an estimator of the mean marginal likelihood lower bound of the full dataset, based on minibatches
[Equation]
where the minibatch X M = { x m } Mm =1 is a randomly drawn sample set of M datapoints from the full dataset X. Such a lower bound also constitutes an important indicator for model evidence in latter experiment.
We call it the empirical variational lower bound (EVLB) in the following.
Note that ˜ L B ( θ, φ, x m ) (cid:39) L ( θ, φ, x m ) and we can deduce that
[Equation]
The last inequality holds due to D KL ( p data ( x ) || p θ ( x )) ≥ 0 .
The alternative optimization strategy can be readily utilized to design the algorithm for solving the model by iteratively updating the noise parameter and the network ones.
During the optimization process, the objective can be monotonically increasing, and thus the algorithm can be guaranteed to be convergent.
The algorithm is summarized as follows:
Optimization for parameters except σ 2 : Optimization for
[Equation]
(Close form solution in regard to ˜ L M ( θ, φ, X M ) .)
Direct gradient method to the transformed variable log simga = log σ ∈ R can be implemented to lift the lower bound ˜
L M as a result to increase the likelihood as well.
8. M O G N OISE D EDUCTION
The noise ε in Eq.
(11) in real situation might be more complex than a simple Gaussian, like that existed in real photographs (Plotz & Roth (2017)).
We thus try to further ameliorate the noise setting as a mixture of Gaussian(MoG) noise.
Such noise modeling strategy has been widely veriﬁed to be effective in applications, like matrix factorization (Meng & Torre (2014)) and robust principal component analysis (Zhao et al.
That is, we assume that
[Equation]
Let c d ∈ { 0 , 1 } K be the latent indicator random one-hot variable, (cid:80)
Kk =1 c dk = 1 , for the MoGnoise component of pixel indexed by d .
Let Π =
[ π 1 , · · · , π K ] and Σ =
[ σ 21 , · · · , σ 2 K ] be the ratio and variance of each component, respectively.
Let Π , Σ be contained in θ .
The conditional joint distribution turns to be
[Equation]
The posterior distribution q φ ( z, c | x ) can be factorized as q φ ( z | x ) q ( c | x, z ) , where q φ ( z | x ) will be direct learnt and the alternative of q ( c | x, z ) , q ( c | x, e ) will be set to the last step p θ ( c | x, e ) in regard to EM procedure.
The lower bound of log p θ ( x ) is then reformulated as follows:
[Equation]
Similar to the Gaussian case, the reparamerization trick is implemented,
[Equation]
where ˜ z =
En ( x ) + Σ z | x 1 / 2 ( x ) e .
By utilizing the SGVB estimator, we get,
[Equation]
Given an input dataset X, we can then construct an estimator to the mean marginal likelihood lower bound of the full dataset, based on minibatches, as follows:
[Equation]
where z m,l =
En ( x m ) + Σ z | x 1 / 2 ( x m ) e ( l ) and the minibatch X M = { x m } Mi =1 is a randomly drawn sample of M datapoints from the full dataset X.
Then let p
[Equation] [Equation]
where z old =
En old ( x ) + Σ oldz | x 1 / 2 ( x ) e , and we can get
[Equation]
The EM algorithm can be naturally employed to solve the model.
The implementation steps are listed as follows:
Step 1.
Expectation Step .
Set
[Equation]
1 , · · · , L .
Calculate the expectation of the latent variable c :
[Equation]
[Equation]
where z m,l : z m,lold =
En old ( x m ) + Σ oldz | x ( x m ) e ( l ) .
The Objective in Maximization Step is obtained as the following,
[Equation]
Step 2.
Maximization Step:
Fix: q ( c | x, e ) determined in the Expectation Step.
[Equation]
Update [Π , Σ] and { θ, φ } / [Π , Σ] by alternative optimization strategy.
Update Π , Σ : note here z m,l : z m,lold =
En old ( x m ) + Σ oldz | x ( x m ) e ( l ) , and we can easily get the closed-form updating formula for these parameters:
[Equation]
x md − G ( z m,lold ) d ) 2 .
Update { θ, φ } / [Π , Σ] : gradient methods with respect to { θ, φ } / [Π , Σ] .
Note here z m,l = En ( x m )+ Σ z | x ( x m ) e ( l ) .
The algorithm can then be summarized as follows:
Initialize the coefﬁcient of { θ, φ } / [Π , Σ] and the coefﬁcient of noise ε : Π , Σ . 2.
Sample e from N (0 , I H ) to obtain e 1 , · · · , e M [One for each element sample in the mini batch in the next step ( L here is set to 1)].
Sample a mini batch X M from p data ( x ) .
Implement EM algorithms as aforementioned (approximate inference for q ( c, z | x ) ): Expectation: calculate γ dmk .
Maximization: update [Π , Σ] , Update { θ, φ } / [Π , Σ] with gradient methods. 5.
Goto 3: Until Trigger End-Criterion.
8. P ROOF
Proof.
For theorem 1.
Proof by Contradiction.
Suppose those two function exist, and we will show that they will be inverse mapping of each other and the homeomorphism mapping of R H and R P .
Since R H and R P have different topology structures ( P (cid:54) = H ), the homeomorphism mapping will not exist.
[Equation]
Since both f and g are continuous, there is a homeomorphism mapping between R H and R P and it leads to the contradiction.
Proof.
For theorem 2.
We only need to test the mean and variance of y .
[Equation]
Therefore [Equation] y is another set of H independent unit Gaussian random variables.
Since x = Q T y , z
and y can generate each other with an linear homeomorphism mapping.
Proof.
For theorem 4,
[Equation]
Proof.
For theorem
[Equation]
8.3. A UXILIARY E XPLANATIONS FOR I NDICATORS
Corollary 1.
The terminology follows the aforemention deﬁnitions and if the involved KLdivergence and mutual information be well deﬁned then
[Equation]
The proof of corollary 1 is the same as that of theorem 5.
This corollary suggests that the estimation in deﬁnition 2 provides another upper bound for the capacity of the encoder network.
Empirically, this estimation is a much tighter estimation than using the estimation in deﬁnition 1.
Corollary 2.
The terminology follows the aforemention deﬁnitions and if the involved KLdivergence and mutual information be well deﬁned then
[Equation]
The corollary is an direct result of theorem 5 and corollary 1.
It suggests that the estimation in deﬁnition 4 is a lower bound for D KL ( q φ ( z )
|| p θ ( z )) .
Deﬁnition 5 (Another Estimation for D KL ( q φ ( z ) ||
p θ ( z )) ) .
[Equation]
Empirically, ¯ D KL ( q φ ( z ) ||
p θ ( z )) and ˜ D KL ( q φ ( z ) ||
p θ ( z )) shown the same estimation results on MNIST.
Deﬁnition 6 (Another estimation for I encoder ( x ; z ) ) .
[Equation]
Deﬁnition 7 (Another estimation for I encoder ( x ; z h ) which quantiﬁes the inﬂuence of each factor) .
[Equation]
8. A NALYSIS ON THE “D ISENTANGLEMENT ” M ETRIC RAISED IN β -VAE (H IGGINS ET AL . (2016))
The terminology inherits those in the β -VAE paper.
The main idea of that “disentanglement” metric is to create a statistic point z diff relevant to the model for each simulated factor respectively and then to use a linear classiﬁer to project the statistic point to the corresponding index of the simulated factor.
If the statistic points induced by the model are easy to be separated then the model is thought to learn “disentangled” representation.
Here, we will argue that even for the idealistic VAE model that follows the disentanglement conditons 3.2.1 could still receive bad score under that performance metric in some situations.
Suppose that the true simulated factors v follows N (0 , I H ) .
Then the learnt “used” 14 factor z can be in the equivalence class [ v ] according to theorem 2.
Concretely, there exists an orthogonal transformation Q such that z
=
Qv .
Suppose that the simulated factors with index y of v is ﬁxed.
Suppose v 1 y − fixed and v 2 y − fixed are two random variable representing the samples from the y -ﬁxed v .
Then the factors inferred by the idealistic VAE turns to be z
1 =
Qv 1 y − fixed and z 2 =
Qv 2 y − fixed .
In order to calculate the statistic point z diff ( y ) =
E | z 1 − z 2 | , we ﬁrst calculate the mean and variance of ( z 1 − z 2 ) .
[Equation]
V ar ( z 1 − z 2 ) =
V ar ( Q ( v 1 y − fixed − v 2 y − fixed ))
=
QCov ( v 1 y − fixed − v 2 y − fixed , v 1 y − fixed − v 2 y − fixed ) Q T = Qdiag (2 , · · · , 2 ,
y , 2 , · · · , 2) Q T =
I − Qdiag (0 , · · · ,
[Equation]
Therefore, z diff ( y ) can be obtained through the diagonal value of 2 I − 2 q y q Ty .
That is,
[Equation]
From the above equation, the location of statistic point is unique determined by ( q 2 y 1 , · · · , q 2 yH ) .
When ( q 2 y 1 , · · · , q 2 yH ) is close to the vertex of the unit cubic for each y then all the statistic points turn to be easily separated.
However, from the perspective of the objective, all the orthogonal Q s are with the same potential to be learnt.
It seems not to be with a small probability that statistic points of different indexes take similar location.
For instance, if H = 2 and Q = (cid:32)
[Equation]
(cid:33) then z diff (1) =
z diff (2) cannot be separated while the representation still follows the disentanglement conditions.
Among different trials, the Q might contribute to that “disentanglement” metric but also might not.
That explains why that metric is unstable.
8. E XPERIMENT D ETAILS
We set L to 1, and minibatch size M to be 100 in all practices.
All the pixels value have been linear normalized in to [0,1].
8.5. C LARIFICATION ON THE CORRECTION ON RESULTS
We ﬁnd in the last version that the code on calculating q ∗ ( z ) is wrong.
Concretely, the objective of the KL-divergence of two Gaussian is incorrect calculated.
That inﬂuences the estimation of
˜ I encoder ( x ; z ) , ˜ I encoder ( x ; z h ) , ˜ D KL ( q φ ( z ) ||
p θ ( z )) and σ 2 z H but has little inﬂuence on the determination of the “used” factors.
So we redo all the experiments on MNIST and delete the relevant results regarding those wrong-calculated indictors on CelebA and Extended Yale Face B.
The new experiment results solve many our past confusions due to the wrong experiment.
They are
• Why σ 2 z h is strongly correlated with ˜ I
( x ; z h ) ?
• Why σ 2 z h of the used factor is always relative small?
• Why can VAE still learn the “disentangled” representation when the learnt ˜ D KL ( q φ ( z ) ||
p θ ( z )) is such big?
Now we know that they are actually not the cases.
Something better is that we ﬁnd our experiment results are more close to our theoretical analysis: when guaranteeing the reconstruction quality in a tolerance range, the smaller ˜ D KL ( q φ ( z ) ||
p θ ( z ))
the less “used” factors are learnt.
That coincide with information conservation theorem: the independent unit Gaussian of the factors assumption facilitates the most efﬁcient coding.
8.5. M NIST
We split randomly 7000 datapoints according to ratio [0 . 6 : 0 . 2 : 0 . 2] into training set, validation set (no use), testing set.
All the indicators and q ∗ ( z ) are evaluated/calculated on 10000 datapoints belonging to the testing set.
All the seed images used to infer latent code and to draw the traversal come from the testing set.
In all ﬁgures of latent code traversal each block corresponds to the traversal of a single latent variable while keeping others ﬁxed to either their inferred ( β -VAE, VAE).
Each row represents a different seed image used to infer the latent values in the VAE-based models.
β -VAE and VAE traversal is over the [ − 3 , 3] range.
The assumed variance σ 2 of noise speciﬁed Gaussian of VAE models is enumerated from [0 . 0005 , 0 . 001 : 0 .
The β setting for the noise learning β -VAE is enumerated from [0 . 1 , 0 . 5 , 1 , 2 : 2 : 18] .
[Figure]
[Figure]
8.5. E XTENDED Y ALE F ACE B
We split randomly 2424 datapoints according to ratio [0 . 8 : 0 . 1 : 0 . 1] into training set, validation set (no use), testing set.
The model is training on the training set.
All the seed images used to infer latent code and to draw the traversal come from the 100 datapoints from the testing set.
In all ﬁgures of latent code traversal each block corresponds to the traversal of a single latent variable while keeping others ﬁxed to either their inferred ( β -VAE, VAE).
Each row represents a different seed image used to infer the latent values in the VAE-based models.
β -VAE and VAE traversal is over the [ − 3 , 3] range.
The β setting for the noise learning β -VAE is enumerated from [1 , 40 , 80 , 120 , 160] .
. C ELEB A
We split randomly roughly 200000 datapoints according to ratio [0 . 8 : 0 . 1 : 0 . 1] into training set, validation set (no use), testing set.
The model is training on the training set.
All the indicators and
I en c ode r ( x ; z h )( na t s )
[Figure]
2 z h
[Figure]
I en c ode r ( x ; z h )( na t s )
[Figure]
2 z h q ∗ ( z ) are evaluated/calculated on 10000 datapoints selected from testing set.
All the seed images used to infer latent code and to draw the traversal come from the 100 datapoints from the testing set.
In all ﬁgures of latent code traversal each block corresponds to the traversal of a single latent variable while keeping others ﬁxed to either their inferred ( β -VAE, VAE).
Each row represents a different seed image used to infer the latent values in the VAE-based models.
β -VAE and VAE traversal is over the [ − 3 , 3] range.
The β setting for the noise learning β -VAE is enumerated from [1 , 30 , 40] .
[Table]
N ETWORK S
[Table]
8. A UXILIARY G ENERATING P ICTURE Note that
only
the
factors
with
˜ I encoder ( x
[Table] [Table]
are
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
= +
Gaussian 1 negative
[Equation]
Gaussian 2 negative σ
[Equation] [Equation]
= 0 .
Gaussian 1 positive σ 22 = 0 .
π 2 = 0 .
Gaussian 2 positive σ 21 = 0 .
π 1 = 0 .