Modeling Text using the Continuous Space Topic Model with Pre-Trained Word Embeddings (2021)

Abstract: In this study, we propose a model that extends the continuous space topic model (CSTM), which flexibly controls word probability in a document, using pre-trained word embeddings. To develop the proposed model, we pre-train word embeddings, which capture the semantics of words and plug them into the CSTM. Intrinsic experimental results show that the proposed model exhibits a superior performance over the CSTM in terms of perplexity and convergence speed. Furthermore, extrinsic experimental results show that the proposed model is useful for a document classification task when compared with the baseline model. We qualitatively show that the latent coordinates obtained by training the proposed model are better than those of the baseline model.

## introduction topic models are statistical models that automatically extract latent topics in documents from a text corpus. topic models have been used in various applications within and outside of natural language processing. such applications include information retrieval @xcite , collaborative filtering @xcite , author identification @xcite , and opinion extraction @xcite . the latent dirichlet allocation (lda) @xcite , which is a representative method for topic modeling, assumes that each document has a latent topic. it uses an unobservable random variable called the latent topic to formulate the factors that produce a set of words that are statistically likely to co-occur. unlike the lda, the continuous space topic model (cstm) @xcite models documents without using intermediate variables, such as latent topics. specifically, the cstm is formulated by introducing latent coordinates of words and considering a function that follows a gaussian process in the same space to represent the importance of a word in a document. in the lda, the probability distribution of words is fixed, and the probability of words is controlled by the topic distribution. therefore, it is not possible to change the probability distribution of words according to each document and thus the text cannot be modeled in a fine-grained way. by contrast, the cstm controls the probability of words based on the latent coordinates of the words and the function representing the meaning of the document. hence, it is possible for the cstm to dynamically change the word distribution according to the document. additionally, the cstm outperforms conventional topic models, such as the lda, in terms of perplexity. as mentioned above, the cstm models documents using word embeddings; however, the structure of the model is such that the word embeddings (latent coordinates) are free parameters. therefore, the estimation of the model is time-consuming because of the large number of parameters. in addition, the only information used for the estimation of the word embeddings is the frequency of words, which makes it difficult to capture the semantics of words. in this study, we propose a new method in which the latent coordinates of words, which are one of the free parameters of the cstm, are learned in advance using word2vec @xcite , and the learned distributed representation of the words are introduced into the cstm. as in the gaussian lda @xcite , when we use the word embeddings that capture the semantics of words and provide them as prior information to the model, we can expect improved performance and faster convergence. in the experiments, we use english and japanese corpora to compare the proposed method with the baseline cstm in terms of perplexity and convergence speed. we also perform a document classification task to evaluate the quality of the document representations that are learned by our model. in the discussion, we use the trained model to investigate the importance of words in documents and evaluate the trained model qualitatively. additionally, we visualize the latent coordinates of words and documents in the same space. the main contributions of this study are as follows: ## conclusion and future work in this study, we introduced the learned distributed representation of words into the cstm to provide prior knowledge on the semantics of words. in the experiments, we showed that the proposed model outperformed the baseline method in terms of perplexity and convergence speed. also, we showed that the proposed model is useful for a document classification task compared with the baseline model. additionally, we showed that the document vectors obtained by training the model are superior through visualization of the embedding space and analysis of importance of words in a document. in the future, we would like to investigate better ways of estimating the model, including optimization by applying the hamiltonian mcmc algorithm, which was not used in this study. furthermore, we would like to use contextualized word embeddings obtained by elmo @xcite or bert @xcite in the proposed model.