Topic Modeling in Embedding Spaces (2020)

Abstract: Topic modeling analyzes documents to learn meaningful patterns of words. However, existing topic models fail to learn interpretable topics when working with large and heavy-tailed vocabularies. To this end, we develop the embedded topic model (etm), a generative model of documents that marries traditional topic models with word embeddings. More specifically, the etm models each word with a categorical distribution whose natural parameter is the inner product between the word’s embedding and an embedding of its assigned topic. To fit the etm, we develop an efficient amortized variational inference algorithm. The etm discovers interpretable topics even with large vocabularies that include rare words and stop words. It outperforms existing document models, such as latent Dirichlet allocation, in terms of both topic quality and predictive performance.

## introduction topic models are statistical tools for discovering the hidden semantic structure in a collection of documents @xcite . topic models and their extensions have been applied to many fields, such as marketing, sociology, political science, and the digital humanities. @xcite provide a review. most topic models build on latent dirichlet allocation (lda) @xcite . lda is a hierarchical probabilistic model that represents each topic as a distribution over terms and represents each document as a mixture of the topics. when fit to a collection of documents, the topics summarize their contents, and the topic figure 1 : ratio of the held-out perplexity on a document completion task and the topic coherence as a function of the vocabulary size for the etm and lda on the 20newsgroup corpus. the perplexity is normalized by the size of the vocabulary. while the performance of lda deteriorates for large vocabularies, the etm maintains good performance. figure 1 illustrates the advantages. this figure shows the ratio between the perplexity on held-out documents (a measure of predictive performance) and the topic coherence (a measure of the quality of the topics), as a function of the size of the vocabulary. (the perplexity has been normalized by the vocabulary size.) this is for a corpus of 11.2k articles from the 20newsgroup and for 100 topics. the red line is lda; its performance deteriorates as the vocabulary size increases-the predictive performance and the quality of the topics get worse. the blue line is the etm; it maintains good performance, even as the vocabulary size become large. like lda, the etm is a generative probabilistic model: each document is a mixture of topics and each observed word is assigned to a particular topic. in contrast to lda, the per-topic conditional probability of a term has a log-linear form that involves a low-dimensional representation of the vocabulary. each term is represented by an embedding and each topic is a point in that embedding space. the topic's distribution over terms is proportional to the exponentiated inner product of the topic's embedding and each term's embedding. figures 2 and 3 show topics from a 300-topic etm of the new york times. the figures show each topic's embedding and its closest words; these topics are about christianity and sports. representing topics as points in the embedding space allows the etm to be robust to the presence of stop words, unlike most topic models. when stop words are included in the vocabulary, the etm assigns topics to the corresponding area of the embedding space (we demonstrate this in section 6). as for most topic models, the posterior of the topic proportions is intractable to compute. we derive an efficient algorithm for approximating the posterior with variational inference @xcite @xcite and additionally use amortized inference to efficiently approximate the topic proportions @xcite . the resulting algorithm fits the etm to large corpora with large vocabularies. this algorithm can either use previously fitted word embeddings, or fit them jointly with the rest of the parameters. (in particular, figures 1 to 3 were made using the version of the etm that uses pre-fitted skip-gram word embeddings.) we compared the performance of the etm to lda, the neural variational document model (nvdm) @xcite , and prodlda @xcite . 1 the nvdm is a form of multinomial matrix factorization and prodlda is a modern version of lda that uses a product of experts to model the distribution over words. we also compare to a document model that combines prodlda with pre-fitted word embeddings. the etm yields better predictive performance, as measured by held-out log-likelihood on a document completion task @xcite . it also discovers more meaningful topics, as measured by topic coherence @xcite and topic diversity. the latter is a metric we introduce in this paper that, together with topic coherence, gives a better indication of the quality of the topics. the etm is especially robust to large vocabularies. ## related work this work develops a new topic model that extends lda. lda has been extended in many ways, and topic modeling has become a subfield of its own. for a review, see @xcite and @xcite . a broader set of related works are neural topic models. these mainly focus on improving topic modeling inference through deep neural networks @xcite @xcite . specifically, these methods reduce the dimension of the text data through amortized inference and the variational auto-encoder (kingma and welling, 2014; @xcite . to perform inference in the etm, we also avail ourselves of amortized inference methods @xcite . as a document model, the etm also relates to works that learn per-document representations as part of an embedding model @xcite @xcite . in contrast to these works, the docu-ment variables in the etm are part of a larger probabilistic topic model. one of the goals in developing the etm is to incorporate word similarity into the topic model, and there is previous research that shares this goal. these methods either modify the topic priors @xcite @xcite or the topic assignment priors @xcite . for example, @xcite use a word similarity graph (as given by a thesaurus) to bias lda towards assigning similar words to similar topics. as another example, @xcite model the perword topic assignments of lda using a markov random field to account for both the topic proportions and the topic assignments of similar words. these methods use word similarity as a type of ''side information'' about language; in contrast, the etm directly models the similarity (via embeddings) in its generative process of words. however, a more closely related set of works directly combine topic modeling and word embeddings. one common strategy is to convert the discrete text into continuous observations of embeddings, and then adapt lda to generate real-valued data @xcite @xcite . with this strategy, topics are gaussian distributions with latent means and covariances, and the likelihood over the embeddings is modeled with a gaussian @xcite or a von-mises fisher distribution @xcite . the etm differs from these approaches in that it is a model of categorical data, one that goes through the embeddings matrix. thus it does not require pre-fitted embeddings and, indeed, can learn embeddings as part of its inference process. the etm also differs from these approaches in that it is amenable to large datasets with large vocabularies. there are few other ways of combining lda and embeddings. @xcite mix the likelihood defined by lda with a log-linear model that uses pre-fitted word embeddings; @xcite randomly replace words drawn from a topic with their embeddings drawn from a gaussian; @xcite adopt a geometric perspective, using wasserstein distances to learn topics and word embeddings jointly; and @xcite propose the neural embedding allocation (nea), which has a similar generative process to the etm but is fit using a pre-fitted lda model as a target distribution. because it requires lda, the nea suffers from the same limitation as lda. these models often lack scalability with respect to the vocabulary size and are fit using gibbs sampling, limiting their scalability to large corpora. ## background the etm builds on two main ideas, lda and word embeddings. consider a corpus of d documents, where the vocabulary contains v distinct terms. let w @xmath0, . . . , v } denote the n th word in the d th document. ## the embedded topic model the etm is a topic model that uses embedding representations of both words and topics. it contains two notions of latent dimension. first, it embeds the vocabulary in an l-dimensional space. these embeddings are similar in spirit to classical word embeddings. second, it represents each document in terms of k latent topics. in traditional topic modeling, each topic is a full distribution over the vocabulary. in the etm, however, the k th topic is a vector α @xmath1. we call α k a topic embeddingit is a distributed representation of the k th topic in the semantic space of words. in its generative process, the etm uses the topic embedding to form a per-topic distribution over the vocabulary. specifically, the etm uses a loglinear model that takes the inner product of the word embedding matrix and the topic embedding. with this form, the etm assigns high probability to a word v in topic k by measuring the agreement between the word's embedding and the topic's embedding. denote the l × v word embedding matrix by ρ; the column ρ v is the embedding of term v. under the etm, the generative process of the d th document is the following: ## inference and estimation we are given a corpus of documents {w 1 , . . . , w d }, where the d th document w d is a collection of n d words. how do we fit the etm to this corpus? the marginal likelihood. the parameters of the etm are the word embeddings ρ 1:v and the topic embeddings α 1:k ; each α k is a point in the word embedding space. we maximize the log marginal likelihood of the documents, the problem is that the marginal likelihood of each document-p(w d | α, ρ)-is intractable to compute. it involves a difficult integral over the topic proportions, which we write in terms of the untransformed proportions δ d in eq. 2, the conditional distribution p(w dn | δ d , α, ρ) of each word marginalizes out the topic assignment z dn , here, θ dk denotes the (transformed) topic proportions (eq. 2) and β k,v denotes a traditional ''topic,'' that is, a distribution over words, induced by the word embeddings ρ and the topic embedding α k , variational inference. we sidestep the intractable integral in eq. eq:integral with variational inference @xcite . variational inference optimizes a sum of perdocument bounds on the log of the marginal likelihood of eq. 4. to begin, posit a family of distributions of the untransformed topic proportions q(δ d ; w d , ν). this family of distributions is parameterized by ν. we use amortized inference, where q(δ d ; w d , ν) (called a variational distribution) depends on both the document w d and shared parameters ν. in particular, q(δ d ; w d , ν) is a gaussian whose mean and variance come from an ''inference network,'' a neural network parameterized by ν (kingma and welling, 2014). the inference network ingests a bag-of-words representation of the document w d and outputs the mean and covariance of δ d . (to accommodate documents of varying length, we form the input of the inference network by normalizing the bag-of-word representation of the document by the number of words n d .) we use this family of distributions to bound the log of the marginal likelihood in eq. 4. the bound is called the evidence lower bound (elbo) and is a function of the model parameters and the variational parameters, the first term of the elbo (eq. 7) encourages variational distributions q(δ d ; w d , ν) that place mass on topic proportions δ d that explain the observed words and the second term encourages q(δ d ; w d , ν) to be close to the prior p(δ d ). maximizing the elbo with respect to the model parameters (α, ρ) is equivalent to maximizing the expected complete log-likelihood, the elbo in eq. 7 is intractable because the expectation is intractable. however, we can form a monte carlo approximation of the elbo, to form an unbiased estimator of the elbo and its gradients, we use the reparameterization trick when sampling the unnormalized proportions δ (1) d , . . . , δ (s) d (kingma and welling, 2014; titsias and lázaro-gredilla, 2014; @xcite . that is, we sample δ (s) we also use data subsampling to handle large collections of documents @xcite . denote by b a minibatch of documents. then the approximation of the elbo using data subsampling is algorithm 1 topic modeling with the etm initialize model and variational parameters for iteration @xmath2, 2, . . . do compute β @xmath3. x d compute µ @xmath4; ν µ ) compute σ @xmath5; ν σ ) sample θ d using eq. 9 and θ @xmath6, ρ, α) = θ ⊤ d β •,w dn end for end for estimate the elbo using eq. 10 and eq. 11 take gradients of the elbo via backpropagation update model parameters α 1:k (ρ if necessary) update variational parameters (ν µ , ν σ ) end for given that the prior p(δ d ) and q(δ d ; w d , ν) are both gaussians, the kl admits a closed-form expression, we optimize the stochastic elbo in equation 10 with respect to both the model parameters (α, ρ) and the variational parameters ν. we set the learning rate with adam (kingma and ba, 2015). the procedure is shown in algorithm 1, where we set the number of monte carlo samples @xmath7; ν) represents a neural network with input x and parameters ν. ## empirical study we study the performance of the etm and compare it to other unsupervised document models. a good document model should provide both coherent patterns of language and an accurate distribution of words, so we measure performance in terms of both predictive accuracy and topic interpretability. we measure accuracy with log-likelihood on a document completion task @xcite ; we measure topic interpretability as a blend of topic coherence and diversity. we find that, of the interpretable models, the etm is the one that provides better predictions and topics. in a separate analysis (section 6.1), we study the robustness of each method in the presence corpora. we study the 20newsgroups corpus and the new york times corpus; the statistics of both corpora are summarized in the 20newsgroup corpus is a collection of newsgroup posts. we preprocess the corpus by filtering stop words, words with document frequency above 70%, and tokenizing. to form the vocabulary, we keep all words that appear in more than a certain number of documents, and we vary the threshold from 100 (a smaller vocabulary, where @xmath8,102) to 2 (a larger vocabulary, where @xmath9,258). after preprocessing, we further remove one-word documents from the validation and test sets. we split the corpus into a training set of 11,260 documents, a test set of 7,532 documents, and a validation set of 100 documents. the new york times corpus is a larger collection of news articles. it contains more than 1.8 million articles, spanning the years 1987-2007. we follow the same preprocessing steps as for 20newsgroups. we form versions of this corpus with vocabularies ranging from @xmath10,842 to @xmath11,237. after preprocessing, we use 85% of the documents for training, 10% for testing, and 5% for validation. ## conclusion we developed the etm, a generative model of documents that marries lda with word embeddings. the etm assumes that topics and words live in the same embedding space, and that words are generated from a categorical distribution whose natural parameter is the inner product of the word embeddings and the embedding of the assigned topic. the etm learns interpretable word embeddings and topics, even in corpora with large vocabularies. we studied the performance of the etm against several document models. the etm learns both coherent patterns of language and an accurate distribution of words.