Improving Contextualized Topic Models with Negative Sampling (2022)

Abstract: Topic modeling has emerged as a dominant method for exploring large document collections. Recent approaches to topic modeling use large contextualized language models and variational autoencoders. In this paper, we propose a negative sampling mechanism for a contextualized topic model to improve the quality of the generated topics. In particular, during model training, we perturb the generated document-topic vector and use a triplet loss to encourage the document reconstructed from the correct document-topic vector to be similar to the input document and dissimilar to the document reconstructed from the perturbed vector. Experiments for different topic counts on three publicly available benchmark datasets show that in most cases, our approach leads to an increase in topic coherence over that of the baselines. Our model also achieves very high topic diversity.

## introduction the modern world is witnessing tremendous growth in digital documents. it is often necessary to organize them into semantic categories to make the content more easily accessible to users. the assignment of domain tags through manual intervention can be quite cumbersome and very expensive to maintain, mainly due to the enormity and diversity of the available data. the use of topic modelling techniques can be of huge significance in this area because of their ability to automatically learn the overarching themes or topics from a collection of documents in an unsupervised way and tag the documents with their dominant topics @xcite @xcite . informally, a topic is a group of extremely related words. while latent dirichlet allocation (lda) @xcite is the classical topic modeling approach, recently neural topic models have become popular as they decouple the inference mechanism from the underlying modeling assumptions (e.g., the topic prior), thereby simplifying the design of new topic models. neural topic models are based on variational autoencoders (vaes) @xcite ) and allow us to leverage the progress in deep learning in modeling text @xcite . the recently proposed contextualized topic model (ctm) @xcite , which is a neural topic model, represents each document in the collection both as a bag-of-words (bow) vector as well as a dense vector produced by a pre-trained transformer like sentence-bert (sbert) @xcite , thus combining a classical representation with a contextualized representation that captures the semantics of the text better. ctm produces state-of-the-art performance on many benchmark datasets @xcite . a neural topic model is trained to maximize the log-likelihood of the reconstruction of the input document and minimize the kl-divergence of the learned distribution of the latent (topic) space from a known prior distribution of the latent space. if the topics in a document are perturbed, that is, say, the top topic in a document is deleted, the document should display a marked change in its word distribution. such an objective is not explicitly modeled above. in this paper, we train ctm to infer topics from a document in such a way that while the inferred topics should aid in reconstructing the document (as in any topic modeling algorithm), when the top topics are perturbed it should fail to reconstruct the original document. this is done by treating the document reconstructed from the correct topic vector as an anchor that is encouraged to be similar to the original input document but dissimilar to the document reconstructed from the perturbed topics. our proposed model, ctm-neg, achieves higher average topic coherence, measured by npmi score, than that of other competing topic models, and very high topic diversity on three datasets. we have made our code publicly avail-able 1 . thus, our primary contributions are: negative samples are produced automatically in an unsupervised way. ## related work latent dirichlet allocation (lda) @xcite models every document in a given corpus as a mixture of topics, where each topic is a probability distribution over the vocabulary. among the modern neural alternatives to lda, a pioneering approach is the prodlda model @xcite . it is a vae-based topic model that uses an approximate dirichlet prior (more precisely, laplace approximation to the dirichlet prior in the softmax basis), instead of a standard gaussian prior @xcite . the vae takes a bag-of-words (bow) representation of a document, maps it to a latent vector using an encoder or inference network, and then maps the vector back to a discrete distribution over words using a decoder or generator network. ctm @xcite augments prodlda by allowing in its input a contextualized representation (sbert) of the documents. embedded topic model (etm) (dieng et al., 2020) is a vae-based topic model that uses distributed representations of both words and topics. negative sampling in nlp-based tasks was popularized after its use in the word embedding model, word2vec @xcite . the idea of negative sampling is to 'sample' examples from 1 https://github.com/adhyasuman/ctmneg a noise distribution and ensure that the model being trained can distinguish between the positive and negative examples. it can be used to reduce the computational cost of training, help identify out-of-distribution examples, or to make the model more robust to adversarial attacks @xcite . a few works have recently applied it to topic modeling. for example, @xcite proposed a negative sampling and quantization model (nqtm) with a modified cross-entropy loss to generate sharper topic distributions from short texts. some researchers have applied generative adversarial networks to design topic models @xcite @xcite , but since the negative examples are generated from an assumed fake distribution, they bear little similarity to real documents. in @xcite , a negative document sample is created by replacing the weights of the words having the highest tf-idf scores in the input document with the weights of the same words in the reconstructed document. our method follows a different strategy: it generates a perturbed document-topic vector (instead of an explicit negative document) and uses triplet loss to push the bow vector reconstructed from the correct topic vector closer to the input bow vector and farther from the bow vector generated from the perturbed topics. unlike the present work, none of the other adversarial topic models use contextual embeddings as input. ## experimental setup we perform all experiments in octis @xcite , which is an integrated framework for topic modeling. ## conclusion we have proposed a negative sampling strategy for a neural contextualized topic model. we evaluated its performance on three publicly available datasets. in most of our experiments, the augmented model achieves higher topic coherence, as measured by npmi and cv, and comparable topic diversity, as captured by irbo, with respect to those of competitive topic models in the literature. a manual evaluation of a few selected topics shows that the topics generated by ctm-neg are indeed coherent and diverse. in the future, we would like to compare it with other contrastive learning-based topic models and integrate it with other neural topic models.