Self-Supervised Neural Topic Modeling (2021)

Abstract: Topic models are useful tools for analyzing and interpreting the main underlying themes of large corpora of text. Most topic models rely on word co-occurrence for computing a topic, i.e., a weighted set of words that together represent a high-level semantic concept. In this paper, we propose a new light-weight Self-Supervised Neural Topic Model (SNTM) that learns a rich context by learning a topic representation jointly from three co-occurring words and a document that the triple originates from. Our experimental results indicate that our proposed neural topic model, SNTM, outperforms previously existing topic models in coherence metrics as well as document clustering accuracy. Moreover, apart from the topic coherence and clustering performance, the proposed neural topic model has a number of advantages, namely, being computationally efficient and easy to train.

## introduction topic models are a means of exploratory document analysis which aim at discovering the underlying themes and narratives within a corpus of text. these models have been extensively used for discovering the latent topical structure of texts in various applications, such as social media analysis @xcite , news analysis @xcite , understanding scientific articles @xcite and more. the most well-known topic model is the latent dirichlet allocation (lda) @xcite , a generative probabilistic model that relies on cooccurrence patterns between observed words to compute latent topics. the inference step of lda is commonly based on approximation methods such as variational inference or collapsed gibbs sampling, due to the intractability of exact inference at scale @xcite . on the other hand, the success of neural word embedding models such as variational auto encoders (kingma and welling, 2014) or word2vec @xcite that also rely on capturing word co-occurrence patterns while using neural network black-box inference opened a new path to neural topic modeling. subsequently, several neural topic models emerged. however, some of these models came with limitations such as: (1) not being able to compute per-document topic distributions, e.g., nvdm @xcite , (2) difficulty of training with respect to computational cost, e.g., topic models based on generative adversarial networks (gans) @xcite . (3) priors based on distributions such as a logistic normal distribution @xcite which may not always model the cooccurrence behaviour among words realistically, as they expect topic proportions of a corpus to follow such patterns. previous research by @xcite ) on word embeddings has shown that a variant of point-wise mutual information (pmi) @xcite computes association patterns among words from a text corpus very similar to that of the word2vec model @xcite even outperforming word2vec on word similarity tasks. this is an indication that pmi is a simple yet effective method for computing word associations and word embeddings. in this paper, we propose the self-supervised neural topic model (sntm) that firstly utilizes the normalized point-wise mutual information (npmi) measure to construct a graph of word connections in order to identify the most prominent words that have the strongest co-occurrence with other words. we show that this method can compute probability scores for words such that the top most probable words are highly similar to those computed using lda. secondly, we design a self-supervised neural network architecture that is trained jointly with top triple word co-occurrences as well as documents that contain them. this approach exposes the neural network to a rich context, i.e., co-occurrence of the triplets with all words from the documents containing the triplet in order to learn topics. this method allows for learning topics from a much richer context information as compared to most other topic models such as lda or even neural variants such as nvlda (srivastava and sutton, 2017) which learn from a single word co-occurrence at a time. here we strive for proposing a different light-weight approach that takes advantage of leveraging multiple word cooccurrences in each training step. therefore, the main contributions of this paper are as follows: the remainder of this paper is organized as follows: in section 2 we present an overview of related work on neural topic modeling. section 3 presents sntm, our novel topic model. we evaluate the topic model in terms of topic coherence and document clustering accuracy on a public dataset in section 4. finally, we conclude the paper in section 5 and present directions of future work. ## related work in this section we review the related work with a focus on neural topic models. one early work on neural topic modeling is the neural variational document model (nvdm) by @xcite which is based on the concept of variational auto encoders. as an unsupervised generative model nvdm is a variational auto encoder consisting of a multi layer perceptron (mlp) encoder that compresses document representations into continuous hidden vectors and a softmax decoder that reconstructs the documents by independently generating the words. a limitation of this model is that it does not explicitly model topic assignments, meaning that per-document topic proportions cannot be computed. the model is also based on a gaussian prior over a hidden state, modeling topics of a document. later, inspired by nvdm, the gaussian softmax model (gsm) @xcite was introduced. as an improvement over nvdm, the gsm modeled topics by providing parameterizable distributions over topics in the framework of neural variational inference. subsequently, another neural topic model, the nvlda @xcite , was proposed. this model is based on an approximation of the dirichlet prior using a logistic-normal distribution. in the past few years, a number of neural topic models based on the generative adversarial networks (gans) @xcite have been introduced. the adversarial-neural topic model (atm) @xcite ) is one such model based on the notion of adversarial training, which however comes at significant additional computational cost. atm models topics using a dirichlet prior which is able to capture topics with multiple distinctive focuses as compared with the logistic-normal prior. another model based on bi-directional gans is the gaussian-bat @xcite which uses a drichlet prior and can infer topic distributions of input documents. additionally, gaussian-bat models a topic using a multivariate gaussian and incorporates the word relatedness into the modeling process. previous work on modeling contexts of words have also used multivariate gaussians @xcite or gaussian mixture models @xcite . finally, another work closely related to ours is the topic modeling method proposed by @xcite . they use the idea of anchor words to model topics. their approach named fastanchor-words performs a distance-based search of farthest words from previously found anchor words. in this work we propose the first self-supervised neural topic modeling method that first uses a novel method for ranking words in terms of importance and association with other words to compute the main seed words upon which topics are formed. second, our proposed model, sntm, goes beyond the basic word co-occurrence methods used in most neural topic models and incorporates a framework for joint learning of co-occurrence of three seed words along with the documents that they originate from in a self-supervised setting. this method allows learning co-occurrence representations from hundreds of words at each step. our empirical results corroborate that our topic model sets a new state-of-the-art performance in terms of document clustering accuracy as well as topic coherence among existing topic models. ## a self-supervised neural topic model in this section we introduce sntm, our proposed neural topic model. in the following subsections we first present a background related to our model, then elaborate on the model architecture. ## evaluation in this section we evaluate sntm by comparing it against several baseline models in terms of topic coherence as well as document clustering accuracy. ## conclusions and future work in this paper, we introduce a novel neural topic model that can learn from co-occurrence patterns between many words at the same time and thus learn coherent topics very efficiently using selfsupervised learning. the model can be trained on commodity hardware and does not require specific architectures such as gpus. we empirically show that our proposed topic model, sntm, sets a new state of the art for topic coherence as well as document clustering accuracy. future work can include further analysis of training data selection methods that may result in improved topic model performance. additionally, we believe that the variable q can be adjusted in order to obtain more fine-grained topics. the same functionality can be exploited to model topic hierarchies in terms of generating topics ranging from generic to highly fine-grained and contextual topics. finally, in this paper we investigated the setting where three co-occurring seed words were used for training the model to expose the model to richer context information. in the future we will further explore other number of word combinations. designing an altogether hierarchical topic model is another potential direction of future work.