A B AYESIAN N ONPARAMETRIC T OPIC M ODEL WITH V ARIATIONAL A UTO -E NCODERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Topic modeling of text documents is one of the most important tasks in representation learning.
In this work, we propose iTM-VAE, which is a Bayesian nonparametric (BNP) topic model with variational auto-encoders.
On one hand, as a BNP topic model, iTM-VAE potentially has inﬁnite topics and can adapt the effective topic number to data automatically.
On the other hand, different with the other BNP topic models, the inference of iTM-VAE is modeled by neural networks, which has rich representation capacity and can be computed in a simple feed-forward manner.
Two variants of iTM-VAE are also proposed in this paper, where iTM-VAE-Prod models the generative process in products-of-experts fashion for better performance and iTM-VAE-G places a prior over the concentration parameter such that the model can adapt a suitable concentration parameter to data automatically.
Experimental results on 20News and Reuters RCV1-V datasets show that the proposed models outperform the state-of-the-arts in terms of perplexity, topic coherence and document retrieval tasks.
Moreover, the ability of adjusting the concentration parameter to data is also conﬁrmed by experiments
I NTRODUCTION
Probabilistic topic models focus on discovering the abstract “topics” that occur in a collection of documents and representing a document as a weighted mixture of the discovered topics.
Classical topic models, the most popular being LDA (Blei et al., 2003), have achieved success in a range of applications, such as information retrieval (Wei & Croft, 2006), document understanding (Blei et al., 2003), computer vision (Rasiwasia & Vasconcelos, 2013) and bioinformatics (Rogers et al., 2005).
A major challenge of topic models is that the inference of the distribution over topics does not have a closed-form solution and must be approximated, using either MCMC sampling or variational inference.
Hence, any small change to the model requires re-designing a new inference method tailored for it.
Moreover, as the model grows more expressive, the inference becomes increasingly complex, which becomes the bottleneck to discover the latent semantic structures of complicated data.
Hence, black-box inference methods (Ranganath et al., 2014; Mnih & Gregor, 2014; Kingma & Welling, 2014b; Rezende et al., 2014), which require only limited knowledge from the models and can be ﬂexibly applied to new models, is desirable for topic models.
Among all the black-box inference methods, Auto-Encoding Variational Bayes (AEVB) (Kingma & Welling, 2014b; Rezende et al., 2014) is a promising one for topic models.
AEVB contains an inference network that can map a document directly to a variational posterior without the need for further local variational updates on test data, and the Stochastic Gradient Variational Bayes (SGVB) estimator allows efﬁcient approximate inference for a broad class of posteriors, which makes topic models more ﬂexible.
Hence, an increasing number of work has been proposed recently to combine topic models with AEVB, such as (Miao et al., 2016; Srivastava & Sutton, 2017; Card et al., 2017;
Miao et al., 2017).
Deciding the number of topics is another challenge for topic models.
One option is to use model selection, which trains models with different topic numbers and selects the best on the validation set.
Bayesian nonparametric (BNP) topic models, however, side-step this issue by allowing the data to determine the number of topics automatically.
For example, Teh et al.
(2006) proposed Hierarchical
Dirichlet Process (HDP), which models each document with a Dirichlet Process (DP) and all DPs for documents in a corpus share a base distribution that is itself also from a DP.
HDP extends LDA in that it can adapt the number of topics to data.
Hence, HDP has potentially an inﬁnite number of topics and allows the number to grow as more documents are observed.
However, the inference of
HDP is more complicated (Blei et al., 2006;
Wang et al., 2011; Teh et al., 2008) and not easy to be applied to new models even with small changes in the generative process.
In this work, we make progress on this problem by proposing an inﬁnite Topic Model with Variational Auto-Encoders (iTM-VAE), which is a Bayesian nonparametric topic model with AEVB.
Coupling Bayesian nonparametric techniques with deep neural networks, iTM-VAE is able to capture the uncertainty regarding to the number of topics, and the inference can be conducted in a simple feed-forward manner.
More speciﬁcally, iTM-VAE uses a stick-breaking process (Sethuraman, 1994) to generate the mixture weights for a countably inﬁnite set of topics, and use neural networks to approximate the variational posteriors.
The main contributions of the paper are:
• We propose iTM-VAE, which, to our best knowledge, is the ﬁrst Bayesian nonparametric topic model equipped with AEVB.
• We propose iTM-VAE-Prod whose distribution over words is a product of experts rather than a mixture of multinomials.
• We propose iTM-VAE-G, which helps the model to adjust the concentration parameter to data automatically.
• The experimental results show that iTM-VAE and its two variants outperform the state-ofthe-art models on two challenging benchmarks signiﬁcantly.
R ELATED W ORK
Topic models have been studied extensively in a variety of applications such as document modeling, information retrieval, computer vision and bioinformatics (Blei et al., 2003; Wei & Croft, 2006;
Putthividhy et al., 2010; Rasiwasia & Vasconcelos, 2013; Rogers et al., 2005).
Please see (Blei, 2012) for an overview.
Recently, with the impressive success of deep learning, neural topic models have been proposed and achieved encouraging performance in document modeling tasks, such as Replicated Softmax (Hinton & Salakhutdinov, 2009), DocNADE (Larochelle & Lauly, 2012), fDARN (Mnih & Gregor, 2014) and NVDM (Miao et al., 2016).
These models achieved competitive performance on modeling documents.
However, they do not explicitly model the generative story of documents, hence are less explainable.
Several recent work has been proposed to model the generative procedure explicitly and the inference of the topic distributions is computed by deep neural networks.
This makes these models interpretable, powerful and easily extendable.
For example, Srivastava & Sutton (2017) proposed
AVITM model, which embeds the original LDA (Blei et al., 2003) formulation with AEVB.
By utilizing Laplacian approximation for the Dirichlet distribution, AVITM can be optimized by the
Stochastic Gradient Variational Bayes (SGVB) (Kingma & Welling, 2014b; Rezende et al., 2014) estimator efﬁciently.
AVITM achieved the state-of-the-art performance on the topic coherence metric (Lau et al., 2014), which indicates the topics learned match closely to human judgment.
The superiority of Bayesian nonparametric topic models, such as HDP (Teh et al., 2006), over the traditional ﬁnite topic models lies in the inﬁnite topic capacity and the self-determined topic number.
However, we do not notice any topic models that combine BNP techniques with deep neural networks.
Actually, Nalisnick & Smyth (2017) proposed Stick-Breaking VAE (SB-VAE), which is a Bayesian nonparameteric version of traditional VAE with a stochastic dimensionality.
However, iTM-VAE differs with SB-VAE in that it is a kind of topic model that models discrete text data.
Furthermore, we also proposed iTM-VAE-G which places a prior on the stick-breaking process such that the model is able to adapt the concentration parameter to data.
Another related work is (Miao et al., 2017), which proposed GSM, GSB, RSB and RSB-TF to model documents.
Particularly, GSB uses a Gaussian stick-breaking construction to model the generative process, which looks similar to the stick-breaking process of iTM-VAE 1 .
However, as commented by Miao et al.
(2017), compared to the stick-breaking process used by iTM-VAE, GSB breaks the exchangeability and is not a Bayesian nonparametric approach.
The RSB-TF from (Miao et al., 2017), which uses a heuristic indicator to guide the growth of the topic numbers, also has the ability to determine the topic number automatically.
However, the performance of RSB-TF does not match its complexity.
Instead, iTM-VAE exploits Bayesian nonparametric techniques to decide the number of topics, which is much more elegant.
And the performance of iTM-VAE also outperforms RSB-TF.
P RELIMINARY
In this section, we brieﬂy describe the stick-breaking process (Sethuraman, 1994; Murphy, 2012),
Variational Auto-Encoders (Kingma & Welling, 2014b; Rezende et al., 2014) and the Kumaraswamy distribution (Kumaraswamy, 1980), which are all essential for iTM-VAE.
3. T HE S TICK -B REAKING P ROCESS
We ﬁrst describe the stick-breaking process, which is used to model the mixture weights over the countably inﬁnite topics for the generative procedure of iTM-VAE.
More speciﬁcally, the stickbreaking process generates an inﬁnite sequence of mixture weights
π = { π k } ∞ k =1 as follows:
[Equation]
This is often denoted as π ∼ GEM ( α ) , where GEM stands for Grifﬁths, Engen and McCloskey (Ewens, 1990) and α is the concentration parameter.
One can see that π k satisﬁes 0 ≤ π k ≤ 1 and (cid:80) ∞ k
=
π k = 1 .
3. V ARIATIONAL A UTO -E NCODERS
Variational Auto-Encoder (VAE) is among the most successful generative models recently.
Speciﬁcally, it assumes that a sample x is generated mathematically as follows:
[Equation]
where p ( z ) is the prior of the latent representation z and p φ ( x | z ) is a conditional distribution with parameters φ to generate x .
The approximation to the posterior of the generative process, q ψ ( z | x ) , is parametrized by an inference neural network with parameters ψ .
According to (Kingma & Welling, 2014b; Rezende et al., 2014), the Evidence Lower Bound (ELBO) of VAE can be written as:
[Equation]
where KL is the Kullback-Leibler divergence.
The parameters φ and ψ can be optimized jointly by applying the Stochastic Gradient Variational
Bayes (SGVB) (Kingma & Welling, 2014b) estimator and Reparameterization Trick (RT).
An essential requirement of SGVB and RT is that the latent variable z can be represented in a differentiable, non-centered parameterization (DNCP)
(Kingma & Welling, 2014a), which allows the gradients to go through the MC expectation.
However, this is not always satisﬁed by the Beta prior of the stick-breaking process for iTM-VAE.
Hence, we resort to the little known Kumaraswamy distribution (Kumaraswamy, 1980) to solve this problem.
3. K UMARASWAMY DISTRIBUTION
Kumaraswamy distribution (Kumaraswamy, 1980) is a continuous distribution, which is mathematically deﬁned as:
[Equation]
where x ∈ [0 , 1] and a, b > 0 .
The inverse cumulative distribution function (CDF) can be expressed in a simple and closed-form formulation, and samples from Kumaraswamy distribution can be drawn by:
[Equation]
Kumaraswamy is similar to Beta distribution, yet more suitable to SGVB since it satisﬁes the DNCP requirement.
Moreover, the KL divergence between Kumaraswamy and Beta can be closely approximated in closed-form.
Hence, we use it to model the inference procedure of iTM-VAE.
T HE M ODEL
In this section, we describe iTM-VAE, a Bayesian nonparametric topic model with VAE.
Speciﬁcally, we ﬁrst describe the generative process of iTM-VAE in Section 4.1, and then the inference process is introduced in Section 4.2.
After that, two variants of the model, iTM-VAE-Prod and iTM-VAE-G, are described in Section 4.3 and Section 4.4, respectively.
4. T HE G ENERATIVE P ROCEDURE
The generative process of iTM-VAE is similar to the original VAE.
The key difference is that the topic distribution (latent variable) π
= { π k } ∞ k =1 is drawn from the GEM distribution to make sure of (cid:80) ∞ k
=
π k = 1 and 0 ≤ π k ≤ 1 .
Speciﬁcally, we suppose each topic θ k = σ ( φ k ) is a probability distribution over vocabulary, where φ k ∈ R V is the parameter of the topic-speciﬁc word distribution, σ ( · ) is the softmax function and V is the vocabulary size.
In iTM-VAE, there are unlimited number of topics and we denote Θ = { θ k } ∞ k =1 and Φ = { φ k } ∞ k =1 as the collections of these countably inﬁnite topics and the corresponding parameters.
The generation of a document by iTM-VAE can then be mathematically described as:
• Draw a topic distribution π ∼ GEM ( α )
Then we get a distribution G ( θ ; π , Θ)
= (cid:80) ∞
k =1 π k δ θ k ( θ ) • For each word w i in the document: 1) draw a topic ˆ θ
i ∼ G ( θ ; π , Θ) ; 2) w i ∼ Cat (ˆ θ i )
where α is a hyper-parameter, Cat
(ˆ θ i ) is a categorical distribution parameterized by ˆ θ i , and δ θ k ( θ ) is a discrete dirac function, which equals to 1 when θ = θ k and 0 otherwise.
Thus, the joint probability of a document with N words w 1: N = { w i } Ni =1 , the topic distribution π and the sampled topics ˆ θ 1: N = { ˆ θ i } Ni =1 can be written as:
[Equation]
where p ( π | α ) =
GEM ( α ) , p ( θ | π , Θ) =
G ( θ ; π , Θ) and p ( w | θ )
=
Cat ( θ ) .
Similar to (Srivastava & Sutton, 2017), we collapse the variable ˆ θ 1: N and rewrite Equation 6 as:
[Equation]
where p ( w i | π , Θ)
=
CAT (¯ θ ) and ¯ θ = (cid:80) ∞
k =1 π k θ k .
In practice, following (Miao et al., 2017), we factorize the parameter φ k of topic θ k as φ k = t k W where t k ∈ R H is the k -th topic factor vector,
W ∈ R H
× V is the word factor matrix and H ∈ R + is the factor dimension.
For simplicity, we still use Φ to denote all the parameters regarding to the generative procedure of iTM-VAE.
Note that, although we parameterize the generative process with Φ , iTM-VAE is still a nonparametric model, since it has potentially inﬁnite model capacity, and can grow the number of parameters with the amount of training data.
4. T HE I NFERENCE P ROCEDURE
In this section, we describe the inference process of iTM-VAE, i.e. how to draw π given a document w 1: N .
To elaborate, suppose ν = [ ν 1 , ν 2 , . .
, ν K − 1 ] is a K − 1 dimensional vector where ν k is a random variable sampled from a Kumaraswamy distribution κ ( ν ; a k , b k ) parameterized by a k and b k , iTM-VAE models the joint distribution q ψ ( ν | w 1: N ) as:
[Equation]
where g ( w 1: N ; ψ ) is a neural network with parameters ψ .
Then, π = { π k } Kk =1 can be drawn by:
[Equation]
In the above procedure, we truncate the inﬁnite sequence of mixture weights π = { π k } ∞ k =1 by K elements, and ν K is always set to 1 to ensure (cid:80)
Kk =
π k = 1 .
Notably, as discussed in (Blei et al., 2006), the truncation of variational posterior does not indicate that we are using a ﬁnite dimensional prior, since we never truncate the GEM prior.
Moreover, the truncation level K is not part of the generative procedure speciﬁcation.
Hence, iTM-VAE still has the ability to model the uncertainty of the number of topics and adapt it to data.
People can manage to use truncation-free posteriors in the model, however, as observed by Nalisnick & Smyth (2017), it does not work well.
On the opposite, the truncated-fashion posterior of iTM-VAE is simple and works well in practice.
iTM-VAE can be optimized by maximizing the Evidence Lower Bound (ELBO):
[Equation]
where we replace Θ with Φ for p ( w 1: N | π , Φ) to emphasize that Φ is the parameter to be optimized, and p ( ν | α ) is products of K − 1 Beta (1 , α ) distributions according to Section 3.1.
The details of the optimization can be found in Appendix 7.1.
4. I TM-VAE WITH P RODUCTS OF E XPERTS
In Equation 7, ¯ θ is a mixture of multinomials.
One drawback of this formulation is that it cannot make any predictions that are sharper than the distributions being mixed, pointed out by (Hinton &
Salakhutdinov, 2009), which may result in some topics that are of poor quality and do not match well with human judgment.
One solution to this issue is to replace the mixture of multinomials with a weighted product of experts which is able to make sharper predictions than any of the constituent experts (Hinton, 2006).
We develop a products-of-experts version of iTM-VAE, which is referred as iTM-VAE-Prod.
Speciﬁcally, we compute a mixed topic distribution ˆ θ = σ ( (cid:80) ∞
k
=1 π k φ k ) for each document, where π k is sampled from GEM ( α ) , and then each word of the document is sampled from Cat (ˆ θ ) .
The beneﬁt of the products-of-experts is demonstrated in Section 5.
4. P LACING A P RIOR ON THE C ONCENTRATION P ARAMETER
In the generative process, the concentration parameter α of GEM ( α ) can have signiﬁcant impact on the growth of number of topics.
The larger the α is, the more “breaks” it will create, and consequently, more topics will be used.
Hence, it is generally reasonable to consider placing a prior on α so that the model can adjust the concentration parameter to data automatically.
Concretely, since the Gamma distribution is conjugate to Beta (1 , α ) , we place a Gamma ( s 1 , s 2 ) prior on α .
Then the ELBO of iTM-VAE can be written as:
[Equation] [Equation]
(13) where p ( α | s 1 , s 2 )
=
Gamma ( s 1 , s 2 ) , p ( v k | α )
=
Beta (1 , α ) , q ( α | γ 1 , γ 2 ) is the variational posterior for α with γ 1 , γ 2 as parameters across the whole dataset.
The derivation for Equation 13 can be found in Appendix 7.2.
In experiments, we ﬁnd iTM-VAE-Prod always performs better than iTMVAE, therefore we only place the prior for iTM-VAE-Prod, and refer this variant as iTM-VAE-G.
E XPERIMENTS
In this section, we evaluate the performance of iTM-VAE and its variants on two public benchmarks: 20News and RCV1-V2.
To make a fair comparison, we use exactly the same data and vocabulary as (Srivastava & Sutton, 2017).
3 We compare iTM-VAE and its variants with several state-of-the-arts, such as DocNADE, NVDM, NVLDA and ProdLDA (Srivastava & Sutton, 2017), GSM, GSB, RSB and RSB-TF, as well as some classical topic models such as LDA and HDP.
The conﬁguration of the experiments is as follows.
We use a two-layer fully-connected neural network for g ( w 1: N ; ψ ) of Equation 8, and the number of hidden units is set to 256 and 512 for 20News and RCV1-V2, respectively.
The factor dimension is set to 200 and 1000 for 20News and RCV1V2, respectively.
The truncation level K in Equation 11 is set to 200 so that the maximum topic numbers will never exceed the ones used by baselines 4 .
Batch-Normalization (Ioffe & Szegedy, 2015) is used to stabilize the training procedure.
The hyper-parameter α for GEM distribution is cross-validated on validation set from [10 , 20 , 30 , 50 , 100] .
We use Adam (Kingma & Ba, 2015) to optimize the model and the learning rate is set to 0 .
01 for all experiments.
The code of iTM-VAE and its variants is available at http://anonymous .
5. P ERPLEXITY E VALUATION
Perplexity is widely used by topic models to measure the goodness-to-ﬁt capability, which is deﬁned as: exp( − 1 D (cid:80) Dd =1 1 | w d | log p ( w d )) , where D is the number of documents , and | w d | is the number of words in the d -th document w d .
Following previous work, the variational lower bound is used to estimate the perplexity.
Table 1 shows the perplexity of different topic models on 20News and RCV1-V2 datasets.
Among these baselines, RSB and GSM (Miao et al., 2017) achieved the lowest perplexities on 20News and
RCV1-V2, which are 785 and 521 , respectively.
While the perplexities achieved by iTM-VAE-Prod on these two benchmarks are 769 and 508 , respectively, which performs better than the state-of-theart.
Moreover, Figure 1-(a) demonstrates perplexities of ﬁnite topic models with different number of topics on 20News.
We can see that, a suitable topic number of these models should be around 10 and 20 .
Interestingly but as expected, the number of effective topics 5 discovered by iTM-VAE is about 19 , which indicates that the Bayesian nonparametric topic model, iTM-VAE, has the ability to determine a suitable number of topics automatically
[Table]
-V2 datasets.
The symbol ” − ” indicates either the model fails to converge within 24 hours, or the original paper does not provide the corresponding values.
We take the best performance of GSM,GSB,RSB and RSB-TF from (Miao et al., 2017).
5. T OPIC C OHERENCE E VALUATION
As the quality of the learned topics is not directly reﬂected by perplexity (Newman et al., 2010), topic coherence is designed to match the human judgment.
As Lau et al.
(2014) showed that Normalized
[Figure]
Pointwise Mutual Information (NPMI) matches the human judgment most closely, we adopt it as the measurement of topic coherence.
Since the number of topics discovered by iTM-VAE is dynamic, we deﬁne the Effective Topic as the topic which becomes the top- 1 signiﬁcant topic of a training sample among the training set more than τ × D times, where D is the number of training samples and τ is a ratio, which is set to 0 .
5% in our experiments.
Following (Miao et al., 2017), we use an average over topic coherence computed by top-5 and topwords for all topics across ﬁve random runs, which is more stable and robust (Lau & Baldwin, 2016).
Table 1 shows the topic coherence of different topic models on 20News and RCV1-V2 datasets.
We can clearly see that iTM-VAE-Prod outperforms all the other topic models signiﬁcantly, which indicates that the topics discovered by iTM-VAE-Prod match more closely to human judgment.
Some topics learned by iTM-VAE-Prod are illustrated in Appendix 7.3.
We also illustrate the topic coherence of ﬁnite topic models with different numbers of topics, and compare them with iTM-VAE-Prod and iTM-VAE on 20News dataset, shown in Figure 1-(b).
The topic coherence of iTM-VAE-Prod outperforms all baselines over all topic numbers.
Another observation is that the best topic coherence of ProdLDA is achieved as the topic number is 15 , which is close to the number of effective topics discovered by iTM-VAE-Prod.
5. D OCUMENT R ETRIEVAL E VALUATION
The document retrieval task is to evaluate the discriminative power of the document representations learned by each model.
We compare iTM-VAE and iTM-VAE-Prod with LDA, DocNADE, NVLDA and ProdLDA.
The setup of the retrieval task is as follows.
The documents in the training/validation sets are used as the database for retrieval, and the test set is used as the query set.
Given a query document, documents in the database are ranked according to the cosine similarities to the query.
Precision/Recall (PR) curves can then be computed by comparing the label of the query with those of the database documents.
For documents who have multiple labels (e.g. RCV1-V2), the PR curves for each of its labels are computed individually and then averaged for each query document.
Finally, the global average of these curves is computed to compare the retrieval performance of each model.
Figure 2 illustrates the PR curves of different models with hidden representations of length 128 .
Speciﬁcally, the mean of the variational Gaussian posterior is used as the features for NVLDA and
ProdLDA.
Since the effective topic numbers of iTM-VAE-Prod and iTM-VAE are dynamic and usually much smaller than 128 on both datasets, we use a weighted sum of topic factor vectors over the topic distributions, where the factor dimension is 128 .
As shown in Figure 2(a) and Figure 2(b), iTM-VAE-Prod always yields competitive results on both datasets, and outperforms the others in most cases.
We also map the latent representations learned by iTM-VAE-Prod to a 2D space by
TSNE (Maaten & Hinton, 2008) and visualize the representations in Figure 2(c).
[Figure]
5. A DAPTING α A UTOMATICALLY BY I TM-VAE-G
As mentioned in Section 4.4, the concentration parameter α of GEM ( α ) has signiﬁcant impact on the growth of the number of topics, which affects the performance of iTM-VAE signiﬁcantly.
As a result, α has to be chosen by cross-validation for better performance.
Figure 3(a) also conﬁrms that the number of effective topics increases with α .
Consequently, iTM-VAE-G, which places a Gamma ( s 1 , s 2 ) prior on α , is proposed to enable the model to adapt α to data automatically.
To verify this, we apply iTM-VAE-G, with a relatively non-informative prior Gamma (1 , 0 . 25) , on several subsets of 20News dataset, which contain 1 , 2 , 5 , 10 and 20 (the whole dataset) classes, respectively.
We initialize the global variational parameters γ 1 and γ 2 of Equation 13 the same as the non-informative prior.
A SGD optimizer with a learning rate of 0 .
01 is used to optimize γ 1 and γ 2 .
Before training, E q ( α | γ 1 ,γ 2 ) [ α ] , the expectation of α given the variational posterior q ( α | γ 1 , γ 2 ) is 4 .
Once the training is done, E q ( α | γ 1 ,γ 2 ) [ α ] will be adjusted to the training set.
Table 2 illustrates γ 1 , γ 2 , E q ( α |
γ 1 ,γ 2 ) [ α ] and the number of effective topics that are learned from data.
We can see that, if the training set contains only 1 class of documents, E q ( α | γ 1 ,γ 2 )
[ α ] will drop to 2 .
35 , and onlyeffective topics are used to model the dataset.
Whereas, when the training set consists of 10 classes of documents, E q ( α |
γ 1 ,γ 2 ) [ α ] increases to 6 . 62 , and 11 effective topics are discovered by the model to explain the dataset.
This indicates that iTM-VAE-G can learn to adjust α to data.
Figure 3(b) illustrates the topic coverage w.r.t the number of topics when the training set contains 1 , 2 , 5 , 10 and 20 classes, respectively.
To this end, we compute the top- 1 signiﬁcant topic for each training document, and sort the topics according to the frequency that it is assigned as top- 1 .
The topic coverage is then deﬁned as the cumulative sum of these frequencies.
Figure 3(b) shows that, with the increasing of the number of classes, more topics are utilized by iTM-VAE-G to reach the same level of topic coverage, which indicates that the model has the ability to adapt to data.
C ONCLUSION
In this paper, we propose iTM-VAE, which, to our best knowledge, is the ﬁrst Bayesian nonparametric topic model that is modeled by Variational Auto-Encoders.
Speciﬁcally, a stick-breaking prior is used to generate the mixture weights of countably inﬁnite topics and the Kumaraswamy distribution is exploited such that the model can be optimized by AEVB algorithm.
Two variants of iTM-VAE are also proposed in this work.
One is iTM-VAE-Prod, which replaces the mixture of multinomials assumption of iTM-VAE with a product of experts for better performance.
The other one is iTMVAE-G which places a Gamma prior on the concentration parameter of the stick-breaking process such that the model can adapt the concentration parameter to data automatically.
The advantage of iTM-VAE and its variants over the other Bayesian nonparametric topics models is that the inference is performed by feed-forward neural networks, which is of rich representation capacity and requires only limited knowledge of the data.
Hence, it is ﬂexible to incorporate more information sources to the model, and we leave it to future work.
Experimental results on two public benchmarks show that iTM-VAE and its variants outperform the state-of-the-art baselines signiﬁcantly.
[Table]
[Figure]
A PPENDIX 7. T HE EVIDENCE LOWER BOUND OF I TM-VAE
In this section we show how to compute the Evidence Lower Bound (ELBO) of iTM-VAE which can be written as:
[Equation]
Similar to other VAE-based models, the SGVB estimator and reparameterization trick can be used to approximate this intractable expectation and propagate the gradient ﬂow into the inference network g .
Speciﬁcally, we have:
[Equation]
where L is the number of Monte Carlo samples in the SGVB estimator and can be set to 1.
N is the number of words in the document.
According to Section 4.2, π ( l ) can be obtained by
[Equation] [Equation]
where g ( w 1: N ; ψ ) is an inference network with parameters ψ , κ denotes the Kumaraswamy distribution and K ∈ R + is the truncation level.
Here we omit the superscript ∗ ( l ) for simplicity.
According to the generative procedure in Section 4.1, p ( w i | π ( l ) , Φ) can be computed by
[Equation]
where t k ∈ R H is the k -th topic factor vector,
W ∈ R H
× V is the word factor matrix, H is the factor dimension, V is the vocabulary size and σ ( · ) is the softmax function.
• KL ( q ψ ( ν | w 1: N ) || p ( ν | α )) :
By applying the KL divergence of a Kumaraswamy distribution κ ( ν ; a k , b k ) from a beta distribution p ( ν ; 1 , α ) , we have:
[Equation]
where B is the Beta function and γ is the Euler’s constant.
7. T HE EVIDENCE LOWER BOUND OF I TM-VAE-G
In this section we show how to compute the Evidence Lower Bound (ELBO) of iTM-VAE-G which can be written as:
[Equation]
Speciﬁcally, each item in Equation 21 can be obtained as follows:
• E q ψ ( ν | w 1: N ) [log p ( w 1: N | π , Φ)] :
The derivation is exactly the same as Appendix 7.1.
• E q ψ ( ν | w 1: N ) q ( α | γ 1 ,γ 2 ) [log p ( ν | α )] :
Recall that the prior of the stick length variable ν k is Beta( 1 , α ): p ( v k | α ) =
α (1 − ν k ) α − 1 and the variational posterior of the concentration parameter α is a gamma distribution q ( α ; γ 1 , γ 2 ) , we have
[Equation]
Now, we provide more details about the calculation of these two expectations in Equation 22 as follows:
[Equation]
Fisrt, we can write the gamma distribution q ( α ; γ 1 , γ 2 ) in its exponential family form:
[Equation]
Considering the general fact that the derivative of the log normalizor log Γ( γ 1 ) − γ 1 log γof a exponential family distribution with respect to its natural parameter γ 1 is equal to the expectation of the sufﬁcient statistic log α , we can compute E q ( α | γ 1 ,
[Equation]
[Equation]
α ] in the ﬁrst term of Equation 22 as follows:
[Equation]
where Ψ is the digamma function, the ﬁrst derivative of the log Gamma function.
◦
[Equation]
By applying the Taylor expansion, E q ( ν k | w 1: N ) [log(1 − v k )] can be written as the inﬁnite sum of the Kumaraswamy’s m th moment:
[Equation]
where B ( · ) is the Beta function.
By substituting Equation 24 and Equation 25 into Equation 22, we can obtain:
[Equation]
w 1: N
[Equation]
log
[Equation]
ψ ( ν
[Equation]
According to Section 4.11 of (Michalowicz et al., 2013), the Kumaraswamy’s entropy is given as
[Equation]
where γ is the Euler’s constant.
• KL ( q ( α | γ 1 , γ 2 ) || p ( α | s 1 , s 2 ))
The KL divergence of one gamma distribution q ( α ; γ 1 , γ 2 ) from another gamma distribution p ( α ; s 1 , s 2 ) evaluates to
[Equation]
7. L EARNED T OPICS
[Table]