D IRICHLET V ARIATIONAL A UTOENCODER
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper proposes Dirichlet Variational Autoencoder (DirVAE) using a Dirichlet prior for a continuous latent variable that exhibits the characteristic of the categorical probabilities.
To infer the parameters of DirVAE, we utilize the stochastic gradient method by approximating the Gamma distribution, which is a component of the Dirichlet distribution, with the inverse Gamma CDF approximation.
Additionally, we reshape the component collapsing issue by investigating two problem sources, which are decoder weight collapsing and latent value collapsing, and we show that DirVAE has no component collapsing; while Gaussian VAE exhibits the decoder weight collapsing and Stick-Breaking VAE shows the latent value collapsing.
The experimental results show that 1) DirVAE models the latent representation result with the best log-likelihood compared to the baselines; and 2) DirVAE produces more interpretable latent values with no collapsing issues which the baseline models suffer from.
Also, we show that the learned latent representation from the DirVAE achieves the best classiﬁcation accuracy in the semi-supervised and the supervised classiﬁcation tasks on MNIST, OMNIGLOT, and SVHN compared to the baseline VAEs.
Finally, we demonstrated that the
DirVAE augmented topic models show better performances in most cases.
I NTRODUCTION
A Variational Autoencoder (VAE) (Kingma & Welling, 2014c) brought success in deep generative models (DGMs) with a Gaussian distribution as a prior distribution (Jiang et al., 2017; Miao et al., 2016; 2017; Srivastava & Sutton, 2017).
If we focus on the VAE, the VAE assumes the prior distribution to be N ( 0 , I ) with the learning on the approximated ˆ µ and ˆ Σ .
Also, Stick-Breaking VAE (SBVAE) (Nalisnick & Smyth, 2017) is a nonparametric version of the VAE, which modeled the latent dimension to be inﬁnite using a stick-breaking process (Ishwaran & James, 2001).
While these VAEs assume that the prior distribution of the latent variables to be continuous random variables, recent studies introduce the approximations on discrete priors with continuous random variables (Jang et al., 2017; Maddison et al., 2017; Rolfe, 2017).
The key of these approximations is enabling the backpropagation with the reparametrization technique, or the stochastic gradient variational Bayes (SGVB) estimator, while the modeled prior follows a discrete distribution.
The applications of these approximations on discrete priors include the prior modeling of a multinomial distribution which is frequently used in the probabilistic graphical models (PGMs).
Inherently, the multinomial distributions can take a Dirichlet distribution as a conjugate prior, and the demands on such prior have motivated the works like Jang et al.
(2017); Maddison et al.
Rolfe (2017) that support the multinomial distribution posterior without explicit modeling on a Dirichlet prior.
When we survey the work with a explicit modeling on the Dirichlet prior, we found a frequent approach such as utilizing a softmax Laplace approximation (Srivastava & Sutton, 2017).
We argue that this approach has a limitation from the multi-modality perspective.
The Dirichlet distribution can exhibit a multi-modal distribution with parameter settings, see Figure 1, which is infeasible to generate with the Gaussian distribution with a softmax function.
Therefore, the previous continuous domain VAEs cannot be a perfect substitute for the direct approximation on the Dirichlet distribution.
Utilizing a Dirichlet distribution as a conjugate prior to a multinomial distribution has an advantage compared to the usage of a softmax function on a Gaussian distribution.
For instance, Figureillustrates the potential difﬁculties in utilizing the softmax function with the Gaussian distribution.
Unlike the Gaussian-Softmax or the GEM distribution, the Dirichlet distribution is able to capture the multi-modality that illustrates multiple peaks at the vertices of the probability simplex.
Given the three-dimensional probability simplex, the Gaussian-Softmax distribution cannot generate the illustrated case of the Dirichlet distribution with a high probability measure at the vertices of the simplex, i.e. the multi-modality where the necessity was emphasized in Hoffman & Johnson (2016).
Additionally, the Grifﬁths-Engen-McCloskey (GEM) distribution (Pitman, 2002), which is the prior distribution of the SBVAE, is difﬁcult to model the multi-modality because the sampling procedure of the GEM distribution is affected by the rich-get-richer phenomenon, so a few components tend to dominate the weight of the samples.
This is different from the Dirichlet distribution that does not exhibit such phenomenon, and the Dirichlet distribution can fairly distribute the weights to the components, and the Dirichlet distribution is more likely to capture the multi-modality by controlling the prior hyper-parameter (Blei et al., 2003).
Then, we conjecture that an enhanced modeling on
Dirichlet prior is still needed 1) because there are cases that the Gaussian-Softmax approaches, or the softmax Laplace approximation, cannot imitate the Dirichlet distribution; and 2) because the nonparametric approaches could be inﬂuenced by the biases that the Dirichlet distribution does not suffer from.
Given these motivations for modeling the Dirichlet distribution with the SGVB estimator, this paper introduces the Dirichlet Variational Autoencoder (DirVAE) that shows the same characteristics of the Dirichlet distribution.
The DirVAE is able to model the multi-modal distribution that was not possible with the Gaussian-Softmax and the GEM approaches.
These characteristics allow the
DirVAE to be the prior of the discrete latent distribution, as the original Dirichlet distribution is.
Introducing the DirVAE requires the conﬁguration of the SGVB estimator on the Dirichlet distribution.
Speciﬁcally, the Dirichlet distribution is a composition of the Gamma random variables, so we approximate the inverse Gamma cumulative distribution function (CDF) with the asymptotic approximation.
This approximation on the inverse Gamma CDF becomes the component of approximating the Dirichlet distribution.
We compared this approach to the previously suggested approximations, i.e. approaches with the Weibull distribution and with the softmax Gaussian distribution, and our approximation shows the best log-likelihood among the compared approximations.
Moreover, we report that we had to investigate the component collapsing along with the research on DirVAE.
It has been known that the component collapsing issue is resolved by the SBVAE because of the meaningful decoder weights from the latent layer to the next layer.
However, we found that SBVAE has latent value collapsing issue resulting in many near-zero values on the latent dimensions that leads to the incomplete utilization of the latent dimension.
Hence, we argue that Gaussian VAE (GVAE) suffers from the decoder weight collapsing , previously limitedly deﬁned as component collapsing ; and SBVAE has a problem of the latent value collapsing .
Finally, we suggest that the deﬁnition of component collapsing should be expanded to represent both cases of decoder weight and latent value collapsings .
The proposed DirVAE shows neither the near-zero decoder weights nor the near-zero latent values, so the reconstruction uses the full latent dimension information in most cases.
We investigated this issue because our performance gain comes from resolving the expanded version of the component collapsing . Due to the component collapsing issues, the existing VAEs have less meaningful latent values or could not effectively use its latent representation.
Meanwhile, DirVAE does not have component collapsing due to the multi-modal prior which possibly leads to superior qualitative and quantitative performances. We experimentally showed that the DirVAE has more meaningful or disentangled latent representation by image generation and latent value visualizations.
Technically, the new approximation provides the closed-form loss function derived from the evidence lower bound (ELBO) of the DirVAE.
The optimization on the ELBO enables the representation learning with the DirVAE, and we test the learned representation from the DirVAE in two folds.
Firstly, we test the representation learning quality by performing the supervised and the semi-supervised classiﬁcation tasks on MNIST, OMNIGLOT, and SVHN.
These classiﬁcation tasks conclude that DirVAE has the best classiﬁcation performances with its learned representation.
Secondly, we test the applicability of DirVAE to the existing models, such as topic models with DirVAE priors on 20Newsgroup and RCV1-v2.
This experiment shows that the augmentation of DirVAE to the existing neural variational topic models improves the perplexity and the topic coherence, and most of best performers were DirVAE augmented.
P RELIMINARIES 2. V ARIATIONAL AUTOENCODERS
A VAE is composed of two parts: a generative sub-model and an inference sub-model.
In the generative part, a probabilistic decoder reproduces ˆ x close to an observation x from a latent variable z ∼ p ( z ) , i.e. x ∼ p θ ( x | z ) =
p θ ( x | ζ ) where ζ = MLP ( z ) is obtained from a latent variable z by a multilayer perceptron (MLP).
In the inference part, a probabilistic encoder outputs a latent variable z ∼ q φ ( z | x ) =
q φ ( z | η ) where η = MLP ( x ) is computed from the observation x by a MLP.
Model parameters, θ and φ , are jointly learned by optimizing the below ELBO with the stochastic gradient method through the backpropagations as the ordinary neural networks by using the SGVB estimators on the random nodes.
[Equation]
In GVAE (Kingma & Welling, 2014c), the prior distribution of p ( z ) is assumed to be a standard
Gaussian distribution.
In SBVAE (Nalisnick & Smyth, 2017), the prior distribution becomes a GEM distribution that produces samples with a Beta distribution and a stick-breaking algorithm.
2. D IRICHLET DISTRIBUTION AS A COMPOSITION OF G AMMA RANDOM VARIABLES
The Dirichlet distribution is a composition of multiple Gamma random variables.
Note that the probability density functions (PDFs) of Dirichlet and Gamma distributions are as follows:
[Equation]
where α k , α, β > 0 .
In detail, if there are K independent random variables following the Gamma distributions
X k ∼ Gamma ( α k , β ) or X ∼ MultiGamma ( α , β · 1 K ) where α k , β > 0 for k = 1 , · · · , K , then we have Y ∼ Dirichlet ( α )
where
Y k
=
X k / (cid:80) X i .
It should be noted that the rate parameter, β , should be the same for every Gamma distribution in the composition.
Then, the KL divergence can be derived as the following:
[Equation]
for P =
MultiGamma ( α , β · 1 K ) and Q =
MultiGamma ( ˆ α , β · 1 K ) where ψ is a digamma function.
The detailed derivation is provided in Appendix B.
2. SGVB FOR G AMMA RANDOM VARIABLE AND APPROXIMATION ON D IRICHLET DISTRIBUTION
This section discusses several ways of approximating the Dirichlet random variable; or the SGVB estimators for the Gamma random variables which compose a Dirichlet distribution.
Utilizing
SGVB requires a differentiable non-centered parametrization (DNCP) for the distribution (Kingma & Welling, 2014d).
The main SGVB for Gamma random variables, used in DirVAE, is using the inverse Gamma CDF approximation explained in the next section.
Prior works include two approaches: the use of the Weibull distribution and the softmax Gaussian distribution, and the two approaches are explained in this section.
Approximation with Weibull distribution.
Because of the similar PDFs between the Weibull distribution and the Gamma distribution, some prior works used the Weibull distribution as a posterior distribution of the prior Gamma distribution (Zhang et al., 2018):
[Equation]
The paper Zhang et al.
(2018) pointed out that there are two useful characteristics when approximating the Gamma distribution with the Weibull distribution.
One useful property is that the KL divergence expressed in a closed form, and the other is the simple reparametrization trick with a closed form of the inverse CDF from the Weibull distribution.
However, we noticed that the Weibull distribution has a component of e − ( x/λ ) k , and the Gamma distribution does not have the additional power term of k in the component.
Since k is placed in the exponential component, small changes on k can cause a signiﬁcant difference that limits the optimization.
Approximation with softmax Gaussian distribution.
As in MacKay (1998); Srivastava & Sutton (2017), a Dirichlet distribution can be approximated by a softmax Gaussian distribution by using a softmax Laplace approximation.
The relation between the Dirichlet parameter α and the Gaussian parameters µ , Σ is explained as the following:
[Equation]
where Σ is assumed to be a diagonal matrix, and we use the reparametrization trick in the usual
GVAE for the SGVB estimator.
M ODEL DESCRIPTION
Along with the inverse Gamma CDF approximation, we describe two sub-models in this section: the generative sub-model and the inference sub-model.
Figure 2 describes the graphical notations of various VAEs and the neural network view of our model. [Equation]
[Figure]
Generative sub-model.
The key difference between the generative models between the DirVAE and the GVAE is the prior distribution assumption on the latent variable z .
Instead of using the standard Gaussian distribution, we use the Dirichlet distribution which is a conjugate prior distribution of the multinomial distribution.
[Equation]
Inference sub-model.
The probabilistic encoder with an approximating posterior distribution q φ ( z | x ) is designed to be Dirichlet ( ˆ α ) .
The approximated posterior parameter ˆ α is derived by the MLP from the observation x with the softplus output function, so the outputs can be positive values constrained by the Dirichlet distribution.
Here, we do not directly sample z from the Dirichlet distribution.
Instead, we use the Gamma composition method described in Section 2.2.
Firstly, we draw v ∼ MultiGamma ( α , β · 1 K ) .
Afterwards, we normalize v with its summation (cid:80) v i .
The objective function to optimize the model parameters, θ and φ , is composed of Equation (1) and (3).
Equation (7) is the loss function to optimize after the composition.
The inverse Gamma CDF method explained in the next paragraph enables the backpropagation ﬂows to the input with the stochastic gradient method.
Here, for the fair comparison of expressing the Dirichlet distribution between the inverse Gamma CDF approximation method and the softmax Gaussian method, we set α k = 1 − 1 /K when µ k = 0 and Σ k = 1 by using Equation (5); and β = 1 .
[Equation]
Approximation with inverse Gamma CDF.
A previous work Knowles (2015) suggested that, if
X ∼ Gamma ( α, β ) , and if F ( x ; α, β ) is a CDF of the random variable X , the inverse CDF can be approximated as F − 1 ( u ; α, β ) ≈ β − 1 ( uα Γ( α )) 1 /α .
Hence, we can introduce an auxiliary variable u ∼ Uniform (0 , 1) to take over all the randomness of X , and we treat the Gamma sampled X as a deterministic value in terms of α and β .
Even though the inverse Gamma CDF method and the decomposition of the Dirichlet distribution into Gamma random variables are found from the past works, to our knowledge, combining the two statistical results is the ﬁrst ﬁnding in the machine learning ﬁeld.
In the next section, we demonstrate that the proposed DirVAE is one of the cornerstones in the ﬁeld of DGMs with various experiments.
E XPERIMENTAL RESULTS
This section reports the experimental results with the following experiment settings: 1) a pure VAE model; 2) a semi-supervised classiﬁcation task with VAEs; 3) a supervised classiﬁcation task with
VAEs; and 4) topic models with DirVAE augmentations.
4. E XPERIMENTS FOR REPRESENTATION LEARNING OF VAE S
Baseline models.
We select the following models as baseline alternatives of the DirVAE: 1) the standard GVAE; 2) the GVAE with softmax (GVAE-Softmax) approximating the Dirichlet distribution with the softmax Gaussian distribution; 3) the SBVAE with the Kumaraswamy distribution (SBVAE-Kuma) & the Gamma composition (SBVAE-Gamma) described in Nalisnick & Smyth (2017); and 4) the DirVAE with the Weibull distribution (DirVAE-Weibull) approximating the
Gamma distribution with the Weibull distribution described in Zhang et al.
We use the following benchmark datasets for the experiments: 1) MNIST; 2) MNIST with rotations (MNIST + rot); 3) OMNIGLOT; and 4) SVHN with PCA transformation.
We provide the details on the datasets in
Appendix D.1.
Experimental setting.
As a pure VAE model, we compare the DirVAE with the following models: GVAE, GVAE-Softmax, SBVAE-Kuma, SBVAE-Gamma, and DirVAE-Weibull.
We use 50dimension and 100 -dimension latent variables for MNIST and OMNIGLOT, respectively.
We provide the details of the network structure and optimization in Appendix D.2.
We set α = 0 .
98 · 1for MNIST and α = 0 .
99 · 1 100 for OMNIGLOT for the fair comparison to GVAEs by using
Equation (5).
All experiments use the Adam optimizer (Kingma & Ba, 2014a) for the parameter learning.
Finally, we acknowledge that the hyper-parameter could be updated as Appendix C, and the experiment result with the update is separately reported in Appendix D.2.
Quantitative result.
For the quantitative comparison among the VAEs, we calculated the MonteCarlo estimation on the marginal negative log-likelihood, the negative ELBO, and the reconstruction loss.
The marginal log-likelihood is approximated as p ( x ) ≈ (cid:80)
i p ( [Equation]
for single instance x where q ( z ) is a posterior distribution of a prior distribution p ( z ) , which is further derived in
Appendix A. Here, we used 100 samples for each 1 , 000 randomly selected from the test data.
Table 1 shows the overall performance of the alternative VAEs.
The DirVAE outperforms all baselines in both datasets from the log-likelihood perspective.
The value of DirVAE comes from the better encoding of the latent variables that can be used for classiﬁcation tasks which we examine in the next experiments.
While the DirVAE-Weibull follows the prior modeling with the Dirichlet distribution, the Weibull based approximation can be improved by adopting the proposed approach with the inverse Gamma CDF. [Table]
Qualitative result.
As a qualitative result, we report the latent dimension-wise reconstructions which are decoder outputs with each one-hot vector in the latent dimension.
Figure 3a showsreconstructed images corresponding to each latent dimension from GVAE-Softmax, SBVAE, and
DirVAE.
We manually ordered the digit-like ﬁgures in the ascending order for GVAE-Softmax and
DirVAE.
We can see that the GVAE-Softmax and the SBVAE have components without signiﬁcant semantic information, which we will discuss further in Section 4.2, and the DirVAE has interpretable latent dimensions in most of the latent dimensions.
Figure 3b also supports the quality of the latent values from DirVAE by visualizing learned latent values through t-SNE (Maaten & Hinton, 2008).
[Figure]
[Figure]
[Figure]
4. D ISCUSSION ON COMPONENT COLLAPSING
Decoder weight collapsing, a.k.a.
component collapsing.
One main issue of GVAE is component collapsing that there are a signiﬁcant number of near-zero decoder weights from the latent neurons to the next decoder neurons.
If these weights become near-zero, the values of the latent dimensions loose inﬂuence to the next decoder, and this means an inefﬁcient learning given a neural network structure.
The same issue occurs when we use the GVAE-Softmax.
We rename this component collapsing phenomenon as decoder weight collapsing to speciﬁcally address the collapsing source.
Latent value collapsing.
SBVAE claims that SBVAE solved the decoder weight collapsing by learning the meaningful weights as shown in Figure 4a.
However, we notice that SBVAE produces the output values, not the weight parameters, from the latent dimension to be near-zero in many latent dimensions after averaging many samples obtained from the test dataset.
Figure 4b shows the properties of DirVAE and SBVAE from the perspective of the latent value collapsing, which SBVAE shows many near-zero average means and near-zero average variances, while DirVAE does not.
The average Fisher kurtosis and average skewness of DirVAE are 5.76 and 2.03, respectively over the dataset, while SBVAE has 20.85 and 4.35, which states that the latent output distribution from SBVAE is more skewed than that of DirVAE.
We found out that these near-zero latent values prevent learning on decoder weights, which we introduce as another type of collapsing problem, as latent value collapsing that is different from the decoder weight collapsing .
These results mean that
SBVAE distributes the non-near-zero latent values sparsely over a few dimensions while DirVAE samples relatively dense latent values.
In other words, DirVAE utilizes the full spectrum of latent dimensions compared to SBVAE, and DirVAE has a better learning capability in the decoder network.
Figure 3a supports the argument on the latent value collapsing by activating each and single latent dimension with a one-hot vector through the decoder.
The non-changing latent dimension-
[Figure]
wise images of SBVAE proves that there were no generation differences between the two differently activated one-hot latent values.
A PPLICATION 1.
EXPERIMENTS OF ( SEMI -) SUPERVISED CLASSIFICATION WITH VAE S
Semi-supervised classiﬁcation task with VAEs.
There is a previous work demonstrating that the
SBVAE outperforms the GVAE in semi-supervised classiﬁcation task (Nalisnick & Smyth, 2017).
The overall model structure for this semi-supervised classiﬁcation task uses a VAE with separate random variables of z and y , which is introduced as the M2 model in the original VAE work (Kingma et al., 2014b).
The detailed settings of the semi-supervised classiﬁcation tasks are enumerated in
Appendix D.3.
Fundamentally, we applied the same experimental settings to GVAE, SBVAE, and
DirVAE in this experiment, as speciﬁed by the authors in Nalisnick & Smyth (2017).
Table 2 enumerates the performances of the GVAE, the SBVAE, and the DirVAE, and the result shows that the error rate of classiﬁcation result using 10% , 5% and 1% of labeled data for each dataset.
In general, the experiment shows that the DirVAE has the best performance out of three alternative VAEs.
Also, it should be noted that the performance of the DirVAE is more improved in the most complex task with the SVHN dataset.
[Table]
Supervised classiﬁcation task with latent values of VAEs.
Also, we tested the performance of the supervised classiﬁcation task with the learned latent representation from the VAEs.
We applied the vanilla version of VAEs to the datasets, and we classiﬁed the latent representation of instances with k -Nearest Neighbor ( k NN) which is one of the simplest classiﬁcation algorithms.
Hence, this experiment can better distinguish the performance of the representation learning in the classiﬁcation task.
Further experimental details can be found in Appendix D.4.
Table 3 enumerates the performances from the experimented VAEs in the datasets of MNIST and
OMNIGLOT.
Both datasets indicated that the DirVAE shows the best performance in reducing the classiﬁcation error, which we conjecture that the performance is gathered from the better representation learning.
It should be noted that, to our knowledge, this is the ﬁrst reported comparison of latent representation learning on VAEs with k NN in the supervised classiﬁcation using OMNIGLOT dataset.
We identiﬁed that the classiﬁcation with OMNIGLOT is difﬁcult given that the k NN error rates with the raw original data are as high as 69 .
41% , and 70 .
This high error rate mainly originates from the number of classiﬁcation categories which is 50 categories in our test setting of OMNIGLOT, compared to 10 categories in MNIST.
[Table]
4. A PPLICATION 2. EXPERIMENTS OF TOPIC MODEL AUGMENTATION WITH D IR VAE
One usefulness of the Dirichlet distribution is being a conjugate prior to the multinomial distribution, so it has been widely used in the ﬁeld of topic modeling, such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003).
Recently, some neural variational topic (or document) models have been suggested, for example, ProdLDA (Srivastava & Sutton, 2017), NVDM (Miao et al., 2016), and
GSM (Miao et al., 2017).
NVDM used the GVAE, and the GSM used the GVAE-Softmax to make the sum-to-one positive topic vectors.
Meanwhile, ProdLDA assume the prior distribution to be the Dirichlet distribution with the softmax Laplace approximation.
To verify the usefulness of the
DirVAE, we replace the probabilistic encoder part of the DirVAE to each model.
Two popular performance measures in the topic model ﬁelds, which are perplexity and topic coherence via normalized pointwise mutual information (NPMI)
(Lau et al., 2014), have been used with 20Newsgroups and RCV1-v2 datasets.
Further details of the experiments can be found in Appendix D.5.
Table 4 indicates that the augmentation of DirVAE improves the performance in general.
Additionally, the best performers from the two measurements are always the experiment cell with DirVAE augmentation except for the perplexity of RCV1-v2, which still remains competent.
[Table]
C ONCLUSION
Recent advances in VAEs have become one of the cornerstones in the ﬁeld of DGMs.
The VAEs infer the parameters of explicitly described latent variables [Table] so the VAEs are easily included in the conventional PGMs.
While this merit has motivated the diverse cases of merging the VAEs to the graphical models [Table] we ask the fundamental quality of utilizing the GVAE where many models have latent values to be categorical probabilities.
The softmax function cannot reproduce the multi-modal distribution that the Dirichlet distribution can.
Recognizing this problem
[Table]
there have been some previous works that approximated the Dirichlet distribution in the VAE settings by utilizing the Weibull distribution or the softmax Gaussian distribution [Table] but the DirVAE with the inverse Gamma CDF shows the better learning performance in our experiments of the representation: the semi-supervised, the supervised classiﬁcations [Table]
and the topic models. Moreover, DirVAE shows no component collapsing and it leads to better latent representation and performance gain.
The proposed DirVAE can be widely used if we recall the popularity of the conjugate relation between the multinomial and the
Dirichlet distributions because the proposed DirVAE can be a brick to the construction of complex probabilistic models with neural networks.
A PPENDIX
This is an appendix for Dirichlet Variational Autoencoder .
Here, we describe the derivations of key equations and experimental setting details which were used in the body of the paper.
The detailed information such as model names, parameter names, or experiment assumptions is based on the main paper.
A M ONTE -C ARLO ESTIMATION ON THE MARGINAL LIKELIHOOD
Proposition A.1.
The marginal log-likelihood is approximated as p ( x ) ≈ (cid:80)
i p ( x | z i ) p ( z i ) q
( z i ) , where q ( z ) is a posterior distribution of a prior distribution p ( z ) .
Proof.
[Equation]
q ( z
[Equation]
) where z i ∼ q ( z )
B KL DIVERGENCE OF TWO MULTI -G AMMA DISTRIBUTIONS
Proposition B.1.
Deﬁne X
=
( X 1 , · · · , X K ) ∼ MultiGamma ( α , β · 1 K ) as a vector of K independent Gamma random variables
X k ∼ Gamma ( α k , β ) where α k , β > 0 for k = 1 , · · · , K .
The KL divergence between two MultiGamma distributions P = MultiGamma ( α , β · 1 K ) and
Q =
MultiGamma ( ˆ α , β · 1 K ) can be derived as the following:
[Equation]
where ψ is a digamma function.
Proof.
Note that the derivative of a Gamma-like function Γ( α ) β α can be derived as follows:
[Equation]
Then
[Equation]
we have the following.
[Equation]
[Equation]
C H YPER - PARAMETER α LEARNING STRATEGY
In this section
[Equation]
we introduce the method of moment estimator (MME) to update the Dirichlet prior parameter α .
Suppose we have a set of sum-to-one proportions D
= { p 1 , · · · , p N } sampled from
Dirichlet ( α ) , then the MME update rule is as the following:
[Equation]
After the burn-in period for stabilizing the neural network parameters
[Equation]
we use the MME for the hyper-parameter learning using the sampled latent values during training.
We alternatively update the neural network parameters and hyper-parameter α .
We choose this estimator because of its closed form nature and consistency (Minka, 2000).
The usefulness of the hyper-parameter update can be found in Appendix D.2.
Proposition C.1.
Given a proportion set D = { p 1 , · · · , p N } sampled from Dirichlet ( α ) , MME of the hyper-parameter α is as the following:
[Equation]
n
[Equation]
k where S = 1 K
[Equation]
k
[Equation]
k for
[Equation]
k
[Equation]
Proof.
Deﬁne µ j,k = E
[ p jk ] as the j th moment of the k th dimension of Dirichlet distribution with prior α .
Then, by the law of large number, µ j,k ≈ ˜ µ j,k .
It can be easily shown that µ 1 ,k = α k (cid:80) i
α i and µ 2 ,k
= α k (cid:80) i α
i
+ α k 1+ (cid:80)
i α i
= µ 1 ,k 1+ α k 1+ (cid:80)
i
α
i so that numerator
[Equation]
denominator
[Equation]
holds for each k = 1 , · · · , K .
Therefore,
[Equation]
and hence,
[Equation]
D E XPERIMENTAL SETTINGS
In this section, we support Section 4 in the original paper with more detailed experimental settings.
Our Tensorﬂow implementation is available at https://TO BE RELEASED .
D. D ATASET DESCRIPTION
We use the following benchmark datasets for the experiments in the original paper: 1) MNIST; 2) MNIST with rotations (MNIST + rot); 3) OMNIGLOT; and 4) SVHN with PCA transformation.
MNIST (LeCun et al., 1998) is a hand-written digit image dataset of size 28 × 28 with 10 labels, consists of 60 , 000 training data and 10 , 000 testing data.
MNIST + rot data is reproduced by the authors of Nalisnick & Smyth (2017) consists of MNIST and rotated MNIST 1 .
OMNIGLOT 2 (Lake et al., 2013; Snderby et al., 2016) is another hand-written image dataset of characters with 28 ×size and 50 labels, consists of 24 , 345 training data and 8 , 070 testing data.
SVHN 3 is a Street
View House Numbers image dataset with the dimension-reduction by PCA into 500 dimensions (Nalisnick & Smyth, 2017).
D. R EPRESENTATION LEARNING OF VAE S
We divided the datasets into { train,valid,test } as the following: MNIST = { 45 , 000 : 5 , 000 : 10 , 000 } and OMNIGLOT = { 22 , 095 : 2 , 250 : 8 , 070 } .
For MNIST, we use 50 -dimension latent variables with two hidden layers in the encoder and one hidden layer in the decoder of 500 dimensions.
We set α = 0 .
98 · 1 50 for the fair comparison to
GVAEs using the Equation (5).
The batch size was set to be 100 .
For OMNIGLOT, we use 100dimension latent variables with two hidden layers in the encoder and one hidden layer in the decoder of 500 dimensions.
We assume α = 0 .
99 · 1 100 for the fair comparison to the GVAEs using the
Equation (5).
The batch size was set to be 15 .
For both datasets, the gradient clipping is used; ReLU function (Nair & Hinton, 2010) is used as an activation function in hidden layers; Xavier initialization (Glorot & Bengio, 2010) is used for the neural network parameter initialization; and the Adam optimizer (Kingma & Ba, 2014a) is used as an optimizer with learning rate 5e-4 for all VAEs except 3e-4 for the SBVAEs.
The prior assumptions for each VAE is the following: 1) N ( 0 , I ) for the GVAE and the GVAE-Softmax; 2) GEM (5) for the SBVAEs; and 3) Dirichlet (0 . 98 · 1 50 ) (MNIST) and Dirichlet (0 . 99 · 1 100 ) (OMNIGLOT) for the DirVAE-Weibull.
We add the result of VAE with 20 normalizing ﬂows (GVAE-NF 20 ) (Rezende & Mohamed, 2015) as a baseline in Table 5.
Also, latent dimension-wise decoder weight norm and t-SNE visualization on latent embeddings of MNIST is given in Figure 5a and 5b which correspond to Figure 4a and 3, respectively.
Additionally, DirVAE-Learning use the same α for the initial value, but the DirVAE-Learning optimizes hyper-parameter α by the following stages through the learning iterations using the MME method in Appendix C: 1) the burn-in period for stabilizing the neural network parameters; 2) the alternative update period for the neural network parameters and α ; and 3) the update period for the neural network parameters with the ﬁxed learned hyper-parameter α .
Table 5 shows that there are improvements in the marginal log-likelihood, ELBO, and reconstruction loss with DirVAE-Learning in both datasets.
We also give the learned hyper-parameter α in Figure 6.
D. S EMI - SUPERVISED CLASSIFICATION TASK WITH VAE S
The overall model structure for this semi-supervised classiﬁcation task uses a VAE with a separate random variable of z and y , which is introduced as the M2 model in the original VAE work (Kingma et al., 2014b).
However, the same task with the SBVAE uses a different model modiﬁed to ignore the relation between the class label variable y and the latent variable z , but they still share the same [Table] [Figure] Learning
[Table] [Figure] parent nodes: q φ ( z , y | x ) = q φ ( z | x ) q φ ( y | x ) where q φ ( y | x ) is a discrimitive network for the unseen labels.
We follow the structure of SBVAE.
Finally, the below are the objective functions to optimize for the labeled and the unlabeled instances of the semi-supervised classiﬁcation task, respectively:
[Equation]
In the above, H is an entropy function.
The actual training on the semi-supervised learning optimizes the weighted sum of Equation (10) and (11) with a ratio hyper-parameter 0 < λ < 1 .
The datasets are divided into { train, valid, test } as the following: MNIST = { 45 , 000 : 5 , 000 : 10 , 000 } , MNIST + rot = { 70 , 000 : 10 , 000 : 20 , 000 } , and SVHN = { 65 , 000 : 8 , 257 : 26 , 032 } .
For SVHN, dimension reduction into 500 dimensions by PCA is applied as preprocessing.
Fundamentally, we applied the same experimental settings to GVAE, SBVAE and DirVAE in this experiment, as speciﬁed by the authors in Nalisnick & Smyth (2017).
4,5 Speciﬁcally, the three
VAEs used the same network structures of 1) a hidden layer of 500 dimension for MNIST; and 2) four hidden layers of 500 dimensions for MNIST + rot and SVHN with the residual network for the last three hidden layers.
The latent variables have 50 dimensions for all settings.
The ratio parameter λ is set to be 0 .
375 for the MNISTs, and 0 .
45 for SVHN.
ReLU function is used as an activation function in hidden layers, and the neural network parameters were initialized by sampling from
N (0 , 0 . 001) .
The Adam optimizer is used with learning rate 3e-4 and the batch size was set to be 100 .
Finally, the DirVAE sets α = 0 .
98 · 1 50 by using Equation (5).
D. S UPERVISED CLASSIFICATION TASK WITH LATENT VALUES OF VAE S
For the supervised classiﬁcation task on the latent representation of the VAEs [Table] we used exactly the same experimental settings as in D.2.
Since DLGMM is basically a Gaussian mixture model with the SBVAE [Table] DLGMM is a more complex model than the VAE alternatives.
We only report the authors’ result from Nalisnick et al.
(2016) for the comparison purposes.
Additionally [Table] we omit the comparison with the VaDE (Jiang et al., 2017) because the VaDE is more customized to be a clustering model rather than the ordinary VAEs that we choose as baselines. [Table]
D. T OPIC MODEL AUGMENTATION WITH D IR VAE
For the topic model augmentation experiment, two popular performance measures in the topic model ﬁelds, which are perplexity and topic coherence via normalized pointwise mutual information (NPMI)
(Lau et al., 2014), have been used with 20Newsgroups 6 and RCV1-v2 7 datasets.
20Newsgroups has 11 , 258 train data and 7 , 487 test data with vocabulary size 1 , 995 .
For the RCV1-v dataset, due to the massive size of the whole data, we randomly sampled 20 , 000 train data and 10 , 000 test data with vocabulary size 10 , 000 .
The lower is better for the perplexity, and the higher is better for the NPMI.
The speciﬁc model structures can be found in the original papers, Srivastava & Sutton (2017);
Miao et al.
We replace the model prior to that of DirVAE to each model and search the hyper-parameter as Table 7 with 1 , 000 randomly selected test data.
We use 500 -dimension hidden layers and 50 topics for 20Newsgroups, and 1 , 000 -dimension hidden layers and 100 topics for RCV1-v2.
Table 8 shows top- 10 high probability words per topic by activating single latent dimensions in the case of 20 Newsgroups.
Also, we visualized the latent embeddings of documents by t-SNE in Figure 7,8, and 9.
[Table]
[Table]
Topicturks turkish armenian genocide village armenia armenians muslims turkey greece
Topicdoctrine jesus god faith christ scripture belief eternal holy bible
Topicseason defensive puck playoff coach score ﬂyers nhl team ice
Topicpitcher braves hitter coach pen defensive injury roger pitch player
Topicide scsi scsus controller motherboard isa cache mb ﬂoppy ram Topictoolkit widget workstation xlib jpeg xt vendor colormap interface pixel Topicspacecraft satellite solar shuttle nasa mission professor lunar orbit rocket Topicknife handgun assault homicide batf criminal gun ﬁrearm police apartment
Topicenforcement privacy encrypt encryption ripem wiretap rsa cipher cryptography escrow
Topicmin detroit tor det calgary rangers leafs montreal philadelphia cal
[Table]
Topicarmenian azerbaijan armenia genocide armenians turkish militia massacre village turks
Topicarab arabs israeli palestinian jews soldier turks nazi massacre jew
Topicresurrection bible christianity doctrine scripture eternal belief christian faith jesus
Topichitter season braves pitcher baseball pitch game player defensive team
Topicdirectory ﬁle compile variable update ftp version site copy host
Topicperformance speed faster mhz rate clock processor average twice fast
Topicwindows microsoft driver dos nt graphic vga card virtual upgrade
Topicseat gear rear tire honda oil front mile wheel engine
Topicpatient disease doctor treatment symptom medical health hospital pain medicine
Topicpt la det tor pit pp vs van cal nj (b) DirVAE augmentation to NVDM
[Table]
turkish armenian armenians people
one
turkey
armenia
[Table] greek history [Table]
israel israeli jews attack world jewish article arab peace land
[Table]
god
[Table]
church christ belief
[Table]
ﬁrst
[Table]
win
[Table]
memory
[Table] ﬁle [Table] server program system ftp
package support
[Table]
center
[Table]
law
[Table]
government right rights
[Table]
police crime
[Table]
[Table]
[Table]
mail
[Table]
fax phone email address information
[Table]