Diversity-Aware Coherence Loss for Improving Neural Topic Models (2023)

Abstract: The standard approach for neural topic modeling uses a variational autoencoder (VAE) framework that jointly minimizes the KL divergence between the estimated posterior and prior, in addition to the reconstruction loss. Since neural topic models are trained by recreating individual input documents, they do not explicitly capture the coherence between words on the corpus level. In this work, we propose a novel diversity-aware coherence loss that encourages the model to learn corpus-level coherence scores while maintaining high diversity between topics. Experimental results on multiple datasets show that our method significantly improves the performance of neural topic models without requiring any pretraining or additional parameters.

## introduction the main goal of topic modeling is to discover latent topics that best explain the observed documents in the corpus. the topics, conceptualized as a multidimensional distribution over the vocabulary, are useful for many downstream applications, including summarization @xcite ), text generation @xcite , dialogue modeling @xcite , as well as analyzing the data used for pretraining large language models @xcite . when presented to humans, they are often represented as lists of the most probable words to assist the users in exploring and understanding the underlying themes in a large collection of documents. while the extrinsic quality of topics can be quantified by the performance of their downstream tasks, the intrinsic interpretability of topics appears to be strongly correlated with two important factors, namely coherence and diversity @xcite . the topic coherence measures to what extent the words within a topic are related to each other in a meaningful way. although human studies provide a direct method for evaluation, they can be costly, especially when a large number of models are waiting to be assessed. therefore, various automatic metrics have been developed to measure topic coherence @xcite @xcite . for instance, the well-established normalized pointwise mutual information (npmi) metric @xcite , based on word co-occurrence within a fixed window, has been found to have a strong correlation with human judgment @xcite . on the other hand, topic diversity measures to what extent the topics are able to capture different aspects of the corpus based on the uniqueness of the topic words @xcite . importantly, studies have shown that optimizing for coherence can come at the expense of diversity @xcite . even without accounting for topic diversity, directly optimizing for topic coherence by itself is a non-trivial task, due to the computational overhead and non-differentiability of the score matrix @xcite . while traditional topic modeling algorithms are in the form of statistical models such as the latent dirichlet allocation (lda) @xcite , advancements in variational inference methods @xcite have led to the rapid development of neural topic model (ntm) architectures @xcite @xcite . more recently, follow-up works have focused on the integration of additional knowledge to improve the coherence of ntms. their attempts include the incorporation of external embeddings @xcite @xcite , knowledge distillation (hoyle et al., 2020), and model pretraining @xcite . however, as the model is designed to operate on a documentlevel input, one significant limitation of ntms is their inability to explicitly capture the corpus-level coherence score, which assesses the extent to which words within specific topics tend to occur together in a comparable context within a given corpus. for example, semantically irrelevant words such as "politics" and "sports" might be contextually relevant in a given corpus (e.g., government funding for the national sports body). recently, one closely related work addresses this gap by reinterpreting topic modeling as a coherence optimization task with diversity as a constraint @xcite . while traditional topic models tend to directly use corpus-level coherence signals, such as factorizing the document-term matrix @xcite , and topic segment labeling with random walks on co-occurrence graphs @xcite , to the best of our knowledge, no existing work have explicitly integrated corpus-level coherence scores into the training of ntms without sacrificing topic diversity. to address this gap, we propose a novel coherence-aware diversity loss, which is effective to improve both the coherence and diversity of ntms by adding as an auxiliary loss during training. experimental results show that this method can significantly improve baseline models without any pretraining or additional parameters foot_0 . ## background latent dirichlet allocation (lda) @xcite is a simple yet effective probabilistic generative model trained on a collection of documents. it is based on the assumption that each document w in the corpus is described by a random mixture of latent topics z sampled from a distribution parameterized by θ, where the topics β are represented as a multidimensional distribution over the vocabulary v . the formal algorithm describing the generative process is presented in appendix a. under this assumption, the marginal likelihood of the document p(w|α, β) is described as: however, since the posterior distribution p(z i |θ) is intractable for exact inference, a wide variety of approximate inference algorithms have been used for lda (e.g., @xcite ). a common strategy to approximate such posterior is employing the variational auto-encoder (vae) @xcite . in particular, ntms use an encoder network to compress the document representation into a continuous latent distribution and pass it to a generative decoder to reconstruct the bag-of-words (bow) representation of the documents. the model is trained to minimize the evidence lower bound (elbo) of the marginal log-likelihood described by the lda generative process: in equation 2, the first term attempts to match the variational posterior over latent variables to the prior, and the second term ensures that the variational posterior favors values of the latent variables that are good at explaining the data (i.e., reconstruction loss). while standard gaussian prior has typically been used in vaes, prodlda @xcite showed that using a laplace approximation of the dirichlet prior achieved superior performance. to further improve topic coherence, combinedtm @xcite concatenated the bow input with contextualized sbert embeddings @xcite , while ze-roshottm @xcite ) used only contextualized embeddings as input. these are the three baselines included in our experiments. ## proposed methodology despite the recent advancements, one significant limitation of the ntm is that since the model is trained on document-level input, it does not have direct access to corpus-level coherence information (i.e., word co-occurrence). specifically, the topicword distribution β is optimized on the documentlevel reconstruction loss, which may not be an accurate estimate of the true corpus distribution due to the inherent stochasticity of gradient-descent algorithms. we address this problem by explicitly integrating a corpus-level coherence metric into the training process of ntms using an auxiliary loss. ## conclusion and future work in this work, we present a novel diversity-aware coherence loss to simultaneously improve the coherence and diversity of neural topic models. in contrast to previous methods, our approach directly integrates corpus-level coherence scores into the training of neural topic models. the extensive experiments show that our proposal significantly improves the performance across all settings without requiring any pretraining or additional parameters. for future work, we plan to perform extensive user studies to examine the extent to which improvements in quantitative metrics affect human preference. further, we would like to extend our approach to other quantitative metrics (e.g., semantic similarity), and perform extrinsic evaluation to study the effects of our approach when the topics are used for downstream tasks (e.g., summarization, dialogue modeling, text generation).