W EAKLY S EMI -S UPERVISED N EURAL T OPIC M ODELS
A BSTRACT
We consider the problem of topic modeling in a weakly semi-supervised setting.
In this scenario, we assume that the user knows a priori a subset of the topics she wants the model to learn and is able to provide a few exemplar documents for those topics.
In addition, while each document may typically consist of multiple topics, we do not assume that the user will identify all its topics exhaustively.
Recent state-of-the-art topic models such as NVDM, referred to herein as Neural
Topic Models (NTMs), fall under the variational autoencoder framework.
We extend NTMs to the weakly semi-supervised setting by using informative priors in the training objective.
After analyzing the effect of informative priors, we propose a simple modiﬁcation of the NVDM model using a logit-normal posterior that we show achieves better alignment to user-desired topics versus other NTM models.
I NTRODUCTION
Topic models are probabilistic models of data that assume an abstract set of topics underlies the data generating process (Blei, 2012).
These abstract topics are often not only useful as feature representations for downstream tasks, but also for exploring and analyzing a corpus.
Topic models are used to explore natural scenes in images (Fei-Fei and Perona, 2005; Mimno, 2012), genetics data (Rogers et al., 2005), and numerous text corpora (Newman et al., 2006; Mimno and McCallum, 2007).
While latent Dirichlet allocation (LDA) serves as the classical benchmark for topic models, recent state-of-the-art topic models such as NVDM (Miao et al., 2016) fall under the variational autoencoder (VAE) framework (Kingma and Welling, 2013), which we refer to as Neural Topic Models (NTMs).
NTMs leverage the ﬂexibility of deep learning to ﬁt an approximate posterior using variational inference.
This posterior can then be used to efﬁciently predict the topics contained in a document.
NTMs have been shown to model documents well, as well as associate a set of meaningful top words with each topic (Miao et al., 2017).
Often, the top words associated with each extracted topic only approximately match the user’s intuition.
Therefore, the user may want to guide the model towards learning topics that better align with natural semantics by providing example documents.
To our knowledge, supervision has been explored in more classical LDA models, but has not been explored yet in the NTM literature.
Labeling the existence of topics in each document across a corpus is prohibitively expensive.
Hence, we focus on a weak form of supervision.
Speciﬁcally, we assume a user may identify the existence of a single topic in a document.
Furthermore, if a user does not specify the existence of a topic, it does not mean the topic does not appear in the document.
The main contribution of our work is an NTM with the ability to leverage minimal user supervision to better align topics to desired semantics.
B ACKGROUND : N EURAL T OPIC M ODELS
Topic models describe documents (typically represented as bags-of-words) as being generated by a mixture of underlying abstract topics.
Each topic is represented as a distribution over the words in a vocabulary so that each document can be expressed as a mixture of different topics and its words drawn from the associated mixture distribution.
A Neural Topic Model is a topic model constructed according to the variational autoencoder (VAE) paradigm.
The generative process for a document x given a latent topic mixture υ is υ ∼ p ( υ ) , x ∼ p θ ( x | υ ) , where p ( υ ) is the prior and θ are learned parameters.
The marginal likelihood p θ ( x ) and posterior p θ ( υ | x ) are in general intractable so a standard approach is to introduce an approximate posterior, q φ ( υ | x ) , and seek maximization of the evidence lower bound (ELBO):
[Equation]
The reparamterization trick (Kingma and Welling, 2013) lets us train this model via stochastic optimization.
The most commonly used reparameterization is for sampling from a Gaussian with diagonal covariance, e.g., υ = µ ( x ) + σ ( x ) (cid:15), (cid:15) ∼ N (0 , 1) where µ ( x ) and σ ( x ) are neural networks.
W EAKLY S EMI -S UPERVISED T OPIC M ODELING
As motivated in the introduction, the topics extracted from unsupervised topic modeling can be challenging to interpret for a user (see Table 1).
It is often the case that a user has a set of “ground truth” topics in mind that they would like the model to align with.
We focus on the setting where a user is presented a subset of documents and identiﬁes at least a single topic for each of the documents. [Figure] harvesting energy chains documents [Equation]
[Figure]
[Table]
We emphasize two challenging components in our setting.
First, it is semi-supervised in the sense that only a small subset of documents is labeled by the user.
Second, the supervision is weak in the sense that the user does not necessarily identify all the topics that are present in a document.
We deﬁne “ground truth” word rankings for each topic using a relative chi-square association metric, a common measure of word association.
Speciﬁcally, we compute the chi-square statistics between the occurrence of each word and occurrence of each document class using the ground truth topic labels.
We then divide the chi-square statistic for each word by the average of its chi-square statistics over all labeled topics, giving us a relative importance that is discriminative across topics.
The top10 words for each topic are then the words with highest relative importance.
To evaluate alignment of topics extracted by an NTM to these “ground truth” topics, we compute the average normalized point-wise mutual information (NPMI) (Bouma, 2009) between each word in the NTM’s topics and each word in the “ground truth” topics—a higher score indicating the NTM’s extracted words often co-occur with the ground truth’s across Wikipedia.
We refer to this score as pairwise-NPMI (P-NPMI).
We additionally report the average NPMI computed between all word pairs within each NTM topic (NPMI) as a measure of topic coherence as well as NPMI separately averaged over just supervised topics (S-NPMI) and unsupervised topics (U-NPMI).
O UR A PPROACH
Our approach focuses on extending the NTM to the semi-supervised setting.
Our method is based on modifying the prior, p i ( υ i ) , of a VAE to incorporate user label information, where p i ( υ i ) is the prior distribution over the latent variables for the i th document.
As discussed above, we assume the user provides partial binary labels indicating the existence of topics in a document (1=exists, 0=unlabeled).
We would like to encourage the posterior samples of the model, υ i ∼ q φ ( υ i | x i ) , to match the true presence of topics in a document.
In the case of a Gaussian posterior (NVDM), a natural interpretation of υ i is as the logits of the probabilities that each topic exists.
Therefore, to recover probabilities, we simply sigmoid the outputs of our encoder.
Given this interpretation for semi-supervision, we set p i ( υ ik ) to be N ( µ ik , 1) , where µ ik = 
 logit (0 . 9999)
if document i is labeled to contain topic k, logit ( p k + ) else if topic k is supervised , logit (
[Equation]
. 5) =
[Equation]
otherwise, (2) and p k + is the class-prior probability, i.e. the probability that any given document contains topic k .
Note that p k + can be estimated from PU data (Jain et al., 2016).
In experiments, we instead use the mean class-prior probability for all topics ( ¯ p + ) and are still able to obtain high performance suggesting this approach is robust to estimation error.
In order to extract topics from the model, we condition the generative distribution, p θ ( x | υ ) , on the logits of each of the possible approximate onehot vectors, e.g., υ 1 = logit ([0 . 9999 , 0 . 0001 , . . . ]) for topic 1.
After conditioning on υ k , we observe the mean of the conditional distribution, ξ k .
We then subtract from each ξ k the mean of ξ over the topics, i.e., ˜ ξ k = ξ k − ¯ ξ .
The indices corresponding to the top-10 values of ˜ ξ k indicate the top-10 words associated with each topic k .
We also explore using two other posteriors.
One is the Concrete / Gumbel-softmax approximation to the Bernoulli (Jang et al., 2016; Maddison et al., 2016) which more closely matches our goal of modeling the existence of topics in documents.
The differentiable posterior q φ ( υ | x ) = Concrete ( π, τ ) can be sampled with Equations (3), (4), and (5):
[Equation]
9999 if document i is labeled to contain topic k, p k + else if topic k is supervised , 0 .
5 otherwise.
(6) where τ is a hyperparameter such that υ ∼ Ber ( π ) as τ → 0 , sigmoid ( r )
= (1 + e − r ) − 1 , and logit ( p ) =
sigmoid − 1 ( p ) =
log( p / (1 − p ) ) .
The prior is p ( υ ik ) ∼ Concrete ( π ik , τ ) where π ik is the prior probability deﬁned in Equation (6).
The log-density of Concrete ( π, τ ) is given in
Section C.3.2 of (Jang et al., 2016) and we use the Bernoulli- D KL as an approximation.
The other posterior we consider is a logit-normal distribution, for which the logit ( υ ) is normally distributed.
Samples are easily obtained by taking the sigmoid of samples from a normal distribution.
C OMPARISON OF K ULLBACK L IEBLER D IVERGENCES
Our technique of incorporating supervision via informative priors provides a supervised training signal via the D KL term in Equation 1.
Figure 1 compares the Bernoulli and Gaussian (NVDM) D KL terms against a standard cross-entropy loss.
Note D KL is the same for Gaussian and logit-normal
In Figure 1, we ﬁx the prior distributions to Ber ( π p =0 . 9999) for the Bernoulli model and N ( µ p = logit (0 . 9999) , σ p =1) for NVDM to signify the presence of a topic with high certainty.
The x-axis, p , denotes the posterior probability: π q for the Bernoulli model and sigmoid ( µ q ) for NVDM/logitnormal.
The cross entropy loss is − π p log p − (1 − π p ) log(1 − p ) where π p is the prior probability.
The posterior variance of the NVDM is assumed to match the prior variance in this ﬁgure.
It is clear that NVDM D KL strictly provides the strongest loss signal for topic prediction.
On the other hand, the Bernoulli model generally achieves better topic coherence (NPMI).
We expect this is due to the cleaner decoding process relative to NVDM.
For example, the Bernoulli decoding computes the word distribution for the ﬁrst out of K topics as the softmax of 1 · W 1 + . . .
+0 · W K + b where W ∈ R | V |× K and b ∈ R | V | are the decoder weight matrix and bias respectively, | V | is the size of the vocabulary, and W k refers to the k th row of W .
In contrast, NVDM decodes with 9 . 2 · W 1 + . . .
− 9 .
2 · W K + b where 9 . 2 ≈ logit (0 . 9999)
= − logit (0 . 0001) .
Therefore, we propose NVDM- σ with a logit-normal posterior.
The intuition is to retain both the strong supervision signal of NVDM as well as the clean decoding used by the Bernoulli model.
E XPERIMENTS
We train and evaluate these models on three multi-label datasets: BibTeX , a tagging dataset containing BibTeX metadata, Delicious , a document classiﬁcation dataset, and AAPD , a dataset extracted from the abstracts and subjects of computer science ArXiv papers.
We vary three experimental variables.
We consider the percentage of topics supervised to be either 10%, 50%, or 100%.
For example, for Delicious with 20 known topics, we consider supervising 2, 10, and all 20 dimensions of υ .
We also consider the number of provided labeled samples per supervised topic.
For example, in one experiment, when supervising 10% of topics for Delicious, we consider providing 3 or 10 example documents per topic.
Lastly, we vary the number of labels given per document.
As described in
Section 3, the user is not required to provide all the topics contained in each document.
We consider documents with only 1 label, up to half the maximum number of labels (5 for Delicious), and fullylabeled (up to 11).
Using 10% topic supervision, 10 labeled documents per topic, and 1 label per document as an experimental baseline, we vary each experimental variable independently resulting in six different settings for each of three datasets.
Note each setting is still weakly semi-supervised.
In general, all models see a rise in NPMI from informative priors.
We observe that NPMI rises on average by 12% relative to the unsupervised setting and by 15% beyond that to the fully supervised case.
This demonstrates that weak labels are capable of providing a signiﬁcant boost in performance.
[Table]
Table 2 shows successful alignment for supervised topics.
Only three documents are labeled for each topic; this positive result is important as labeling numerous documents is prohibitively expensive for commercial applications.
Table 3 shows NVDM- σ does best in terms of all NPMI metrics.
[Table]
C ONCLUSION
In this work
[Table]
we proposed supervising Neural Topic Models with weak supervision via informative priors and explored a variety of model posteriors.
A careful analysis of their KL divergences and decoding mechanisms led us to an NTM with logit-normal posterior which best aligned extracted topics to desired user semantics. A M ODEL ARCHITECTURE
We use a feedforward neural network with 2 hidden layers followed by a linear transformation for the encoder.
The ﬁrst layer is 3 K neurons wide and the second is 2 K neurons wide where K is the size given to the latent space.
Each hidden layer uses sigmoid nonlinearities.
The ﬁnal layer is linear and outputs a vector of size 2 K for the parameters of the Gaussian posterior.
The mean is obtained from the ﬁrst K entries of this vector and the standard deviation, σ , is obtained from the last K entries by applying a softplus, log(1 + e x ) .
Instead of directly modeling the probability π in Equation (5) for the Bernoulli posterior, we interpret the output of the ﬁnal linear layer of the encoder as logit ( π ) .
We set the dimensionality of the latent-space υ equal to 50 for 20NG and the number of ground truth topics for all other datasets.
The decoder always consists of a linear layer followed by a softmax to obtain the multinomial probabilities for each word in the vocabulary.
We set τ = 1 for the Gumbel-softmax sampler.
We train with Adadelta, rescale gradients by 0.
(smaller), and use batch sizes of 256.
B S EMI -S UPERVISED T OPIC M ODEL T EST P ERFORMANCE
Metric
Model
Bibtex
Delicious
ArXiv
PPX
NVDM
Bernoulli
NVDM+sig   − log( p ( x | υ ))
NVDM
Bernoulli
NVDM+sig
P-NPMI
NVDM 0.
Bernoulli 0.
NVDM+sig 0.
NPMI (Avg/S)
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]