A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings (2020)

Abstract: We propose a novel generative model to explore both local and global context for joint learning topics and topic-specific word embeddings. In particular, we assume that global latent topics are shared across documents, a word is generated by a hidden semantic vector encoding its contextual semantic meaning, and its context words are generated conditional on both the hidden semantic vector and global latent topics. Topics are trained jointly with the word embeddings. The trained model maps words to topic-dependent embeddings, which naturally addresses the issue of word polysemy. Experimental results show that the proposed model outperforms the word-level embedding methods in both word similarity evaluation and word sense disambiguation. Furthermore, the model also extracts more coherent topics compared with existing neural topic models or other models for joint learning of topics and word embeddings. Finally, the model can be easily integrated with existing deep contextualized word embedding learning methods to further improve the performance of downstream tasks such as sentiment classification.

## introduction probabilistic topic models assume that words are generated from latent topics that can be inferred from word co-occurrence patterns taking a document as global context. in recent years, various neural topic models have been proposed. some of them are built on the variational auto-encoder (vae) @xcite , which utilizes deep neural networks to approximate the in-tractable posterior distribution of observed words given latent topics @xcite @xcite . however, these models take the bag-of-words (bows) representation of a given document as the input to the vae and aim to learn hidden topics that can be used to reconstruct the original document. they do not learn word embeddings concurrently. other topic modeling approaches explore the pre-trained word embeddings for the extraction of more semantically coherent topics since word embeddings capture syntactic and semantic regularities by encoding the local context of word co-occurrence patterns. for example, the topicword generation process in the traditional topic models can be replaced by generating word embeddings given latent topics @xcite or by a two-component mixture of a dirichlet multinomial component and a word embedding component @xcite . alternatively, the information derived from word embeddings can be used to promote semantically related words in the polya urn sampling process of topic models @xcite or generate topic hierarchies @xcite . however, all these models use pretrained word embeddings and do not learn word embeddings jointly with topics. word embeddings could improve the topic modeling results, but conversely, the topic information could also benefit word embedding learning. early word embedding learning methods @xcite learn a mapping function to project a word to a single vector in an embedding space. such one-to-one mapping cannot deal with word polysemy, as a word could have multiple meanings depending on its context. for example, the word 'patient' has two possible meanings 'enduring trying circumstances with even temper' and 'a person who requires medical care'. when analyzing reviews about restaurants and health services, the semantic meaning of 'patient' could be inferred depending on which topic it is associated with. one solution is to first extract topics using the standard latent dirichlet allocation (lda) model and then incorporate the topical information into word embedding learning by treating each topic as a pseudo-word @xcite . whereas the aforementioned approaches adopt a two-step process, by either using pre-trained word embeddings to improve the topic extraction results in topic modeling, or incorporating topics extracted using a standard topic model into word embedding learning, @xcite developed a skip-gram based model to jointly learn topics and word embeddings based on the probabilistic latent semantic analysis (plsa), where each word is associated with two matrices rather than a vector to induce topic-dependent embeddings. this is a rather cumbersome setup. @xcite used the skip-gram to imitate the probabilistic topic model that each word is represented as an importance vector over topics for context generation. in this paper, we propose a neural generative model built on vae, called the joint topic wordembedding (jtw) model, for jointly learning topics and topic-specific word embeddings. more concretely, we introduce topics as tangible parameters that are shared across all the context windows. we assume that the pivot word is generated by the hidden semantics encoding the local context where it occurred. then the hidden semantics is transformed to a topical distribution taking into account the global topics, and this enables the generation of context words. our rationale is that the context words are generated by the hidden semantics of the pivot word together with a global topic matrix, which captures the notion that the word has multiple meanings that should be shared across the corpus. we are thus able to learn topics and generate topic-dependent word embeddings jointly. the results of our model also allow the visualization of word semantics because topics can be visualized via the top words and words can be encoded as distributions over the topics 1 . 1 our source code is made available at http:// github.com/somethingx02/topical_wordvec_ models. in summary, our contribution is three-fold: our work is related to two lines of research: skip-gram approaches for word embedding learning. the skip-gram, also known as word2vec @xcite , maximizes the probability of the context words w n given a centroid word x n . @xcite pointed out that skip-gram neglects the global word cooccurrence statistics. they thus formulated the skip-gram as a non-negative matrix factorization (nmf) with the cross-entropy loss switched to the least square error. another nmf-based method was proposed by @xcite , in which the euclidean distance was substituted with wasserstein distance. @xcite rewrote the nmf objective as a cumulative product of normal distributions, in which each factor is multiplied by a von mises-fisher (vmf) distribution of context word vectors, to hopefully cluster the context words since the vmf density retains the cosine similarity. although the skip-gram-based methods attracted extensive attention, they were criticized for their inability to capture polysemy @xcite . a pioneered solution to this problem is the multiple-sense skip-gram model @xcite , where word vectors in a context are first averaged then clustered with other contexts to obtain a sense representation for the pivot word. in the same vein, @xcite leveraged sense tags annotated by babelnet @xcite to jointly learn word and sense representations in the skip-gram manner that the context words are parameterized via a shared look-up table and sent to a bilstm to match the pivot word vector. there have also been bayesian extensions of the skip-gram models for word embedding learning. @xcite inherited the probabilistic generative line while extending the skip-gram by placing a gaussian prior on the parameterized word vectors. the parameters were estimated via variational inference. in a similar vein, @xcite proposed to generate words in bilingual parallel sentences by shared hidden semantics. they introduced a latent index variable to align the hidden semantics of a word in the source language to its equivalence in the target language. more recently, @xcite proposed the bayesian skip-gram (bsg) model, in which each word type with its related word senses collapsed is associated with a 'prior' or static embedding and then, depending on the context, the representation of each word is updated by 'posterior' or dynamic embedding. through bayesian modeling, bsg is able to learn context-dependent word embeddings. it does not explicitly model topics, however. in our proposed jtw, global topics are shared among all documents and learned from data. also, whereas bsg only models the generation of context words given a pivot word, jtw explicitly models the generation of both the pivot word and the context words with different generative routes. combining word embeddings with topic modeling. pre-trained word embeddings can be used to improve the topic modeling performance. for example, @xcite proposed the gaussian lda model, which, instead of generating discrete word tokens given latent topics, generates draws from a multivariate gaussian of word embeddings. nguyen et al. (2015) also replaced the topic-word dirichlet multinomial component in traditional topic models, but by a two-component mixture of a dirichlet multinomial component and a word embedding component. @xcite proposed to modify the polya urn sampling process of the lda model by promoting semantically related words obtained from word embeddings. more recently, @xcite proposed to adapt a multi-layer gamma belief network to generate topic hierarchies and also fine-grained interpretation of local topics, both of which are informed by word embeddings. instead of using word embeddings for topic modeling, @xcite proposed the topical word embedding model, which incorporates the topical information derived from standard topic models into word embedding learning by treating each topic as a pseudo-word. @xcite followed this route and proposed a four-stage model in which topics were first extracted from a corpus by lda and then the topic-based word embeddings are mapped to a shared space using anchor words that were retrieved from the wordnet. there are also approaches proposed to jointly learn topics and word embeddings built on skip-gram models. @xcite developed a skip-gram topical word embedding (ste) model built on plsa where each word is associated with two matrices-one matrix used when the word is a pivot word and another used when the word is considered as a context word. expectation-maximization is used to estimate model parameters. @xcite proposed the mixed-membership skip-gram model (mmsg), which assumes a topic is drawn for each context and the word in the context is drawn from the logbilinear model based on the topic embeddings. foulds trained their model by alternating between gibbs sampling and noise-contrastive estimation. mmsg only models the generation of context words, but not pivot words. whereas our proposed jtw also resembles the similarity to the skip-gram model in that it predicts the context word given the pivot word, it is different from the existing approaches in that it assumes global latent topics shared across all documents and the generation of the pivot word and the context words follows different generative routes. moreover, it is built on vae and is trained using neural networks for more efficient parameter inference. following the problem setup in the skip-gram model, we consider a pivot word x n and its context window w @xmath0,1:c . we assume there are a total of n pivot word tokens and each context window contains c context words. however, as opposed to skip-gram, we do not compute the joint probability as a product chain of conditional probabilities of the context word given the pivot. instead, in our model, context words are represented as bows for each context window by assuming the exchangeability of context words within the local context window. we hypothesize that the hidden semantic vector z n of each word x n induces a topical distribution that is combined with the global corpus-wide latent topics to generate context words. topics are represented as a probability matrix where each row is a multinomial distribution measuring the importance of each word within a topic. the hidden semantics z n of the pivot word x n is transformed to a topical distribution ζ n , which participates in the generation of context words. our assumption is that each word embodies a finite set of meanings that can be interpreted as topics, thus each word representation can be transformed to a distribution over topics. context words are generated by first selecting a topic and then sampled according to the corresponding multinomial distribution. this enables a quick understanding of word semantics through the topical distribution and at the same time learning the latent topics from the corpus. the generative process is given below: here, all the distributions are functions approximated by neural networks, e.g., p(x n |z n ) ∝ exp (m x z n + b x ), which will be discussed in more details in the decoder section, t n,c indexes a row β t n,c in the topic matrix. we could implicitly marginalize out the topic indicators, in which case the probability of a word would be written as w n,c |ζ n , β ∼ categorical (σ(β t ζ n )), where σ(•) denotes the softmax function. the prior distribution for z n is a multivariate gaussian distribution with the mean 0 and covariance i, of which the posterior indicates the hidden semantics of the pivot word when conditioned on {x n , w n }. although both jtw and bsg assume that a word can have multiple senses and use a latent embedding z to represent the hidden semantic meaning of each pivot word, there are some key differences in their generative processes. jtw first draws a latent embedding z from a standard gaussian prior that is deterministically transformed into topic distributions and a distribution over pivot words. the pivot word is conditionally independent of its context given the latent embedding. at the same time, each context word is assigned a latent topic, drawn from a shared topic distribution which leverages the global topic information, and then drawn independently of one another. in bsg the latent embedding z is also drawn from a gaussian prior but the context words are generated directly from the latent embedding z, as opposed to via a mixture model as in jtw. therefore, jtw is able to group semantically similar words into topics, which is not the case in bsg. given the observed variables {x 1:n , w 1:n }, the objective of the model is to infer the posterior p(z|x, w). this is achieved by the vae framework. as illustrated in figure 1 , the jtw model is composed of an encoder and a decoder, each of which is constructed by neural networks. the family of distributions to approximate the posterior is gaussian, in which µ n and σ n are optimized. as in vae, we optimize µ n and σ n through the training of parameters in neural networks (e.g., we optimize m π in µ @xmath1. ## experimental setup dataset. we train the proposed jtw model on the yelp dataset, foot_1 which is a collection of more than 4 million reviews on over 140k business categories. although the number of business categories is large, the vast majority of reviews falls into 5 business categories. the top restaurant category consists of more than 40% of reviews. the next top 4 categories, shopping, beauty &spas, automotive, and clinical, contain about 8%, 6%, 4%, and 3% of reviews, respectively. the clinical documents are further filtered by business subcategories defined in @xcite , which are recognized as core clinical businesses. this results in 176,733 documents for the clinical category. because the dataset is extremely imbalanced, simply training the model on the original dataset will likely overfit to the restaurant category. we thus balance the dataset by sampling roughly an equal number of documents from each of the top 5 categories. the vocabulary size is set to 8,000. we use mallet foot_2 to filter out stopwords. the final dataset consists of 865,616 documents with a total of 101,468,071 tokens. parameter setting. the word semantics are represented as 100-dimensional vectors (i.e., @xmath2, which is a default configuration for word representations @xcite . the number of latent topics is set to 50. it has been previously studied in kingma and welling (2014) that the number of samples per data point can be set to 1 if the batch size is large, (e.g., > 100). in our experiments, we set the batch size to 2,048 and the number of samples per data point, s, to 1. the context window size is set to 10. network parameters (i.e., θ, φ) are all initialized by a normal distribution with zero mean and 0.1 variance. baselines. we compare our model against four baselines: among the aforementioned baselines, cvmf and bsg only generate word embeddings and do not model topics explicitly. also, cvmf only maps each word to a single word embedding whereas bsg can output context-dependent word embeddings. both ste and mmsg can learn topics and topic-dependent embeddings at the same time. however, in ste the topic dependence is stored in the lines of word matrices and the word representations themselves are context independent. in contrast, mmsg associates each word with a topic distribution; it could produce contextualized word embeddings by summing up topic vectors weighed by the posterior topic distribution given a context. we probe into different topic counts and find the best setting for methods with topics or mixtures. in all the baselines, the dimensionality of word embeddings is tuned and finally set to 100. ## experimental results we compare jtw with baselines on both word similarity and word-sense disambiguation tasks for the learned word embeddings, and also present the topic coherence and qualitative evaluation results for the extracted topics. furthermore, we show that jtw can be easily integrated with deep contextualized word embeddings to further improve the performance of downstream tasks such as sentiment classification. ## conclusion driven by the motivation that combining word embedding learning and topic modeling can mutually benefit each other, we propose a probabilistic generative framework that can jointly discover more semantically coherent latent topics from the global context and also learn topic-specific word embeddings, which naturally address the problem of word polysemy. experimental results verify the effectiveness of the model on word similarity evaluation and word sense disambiguation. furthermore, the model can discover latent topics shared across documents, and the encoder of jtw can generate the topical distribution for each word. this enables an intuitive understanding of word semantics. we have also shown that our proposed jtw can be easily integrated with deep contextualized word embeddings to further improve the performance of downstream tasks. in future work, we will explore the discourse relationships between context windows to model, for example, the semantic shift between the neighboring sentences. the university of warwick. dz was partially funded by the national key research and development program of china (2017yfb1002801) and the national natural science foundation of china (61772132).