Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2021)

Abstract: Topic models extract groups of words from documents, whose interpretation as a topic hopefully allows for a better understanding of the data. However, the resulting word groups are often not coherent, making them harder to interpret. Recently, neural topic models have shown improvements in overall coherence. Concurrently, contextual embeddings have advanced the state of the art of neural models in general. In this paper, we combine contextualized representations with neural topic models. We find that our approach produces more meaningful and coherent topics than traditional bag-of-words topic models and recent neural models. Our results indicate that future improvements in language models will translate into better topic models.

## introduction one of the crucial issues with topic models is the quality of the topics they discover. coherent topics are easier to interpret and are considered more meaningful. e.g., a topic represented by the words "apple, pear, lemon, banana, kiwi" would be considered a meaningful topic on fruit and is more coherent than one defined by "apple, knife, lemon, banana, spoon." coherence can be measured in numerous ways, from human evaluation via intrusion tests @xcite to approximated scores @xcite . however, most topic models still use bag-of-words (bow) document representations as input. these representations, though, disregard the syntactic and semantic relationships among the words in a document, the two main linguistic avenues to coherent text. i.e., bow models represent the input in an inherently incoherent manner. meanwhile, pre-trained language models are becoming ubiquitous in natural language processing (nlp), precisely for their ability to cap-ture and maintain sentential coherence. bidirectional encoder representations from transformers (bert) @xcite , the most prominent architecture in this category, allows us to extract pre-trained word and sentence representations. their use as input has advanced state-of-the-art performance across many tasks. consequently, bert representations are used in a diverse set of nlp applications @xcite . various extensions of topic models incorporate several types of information @xcite @xcite , use word relationships derived from external knowledge bases @xcite @xcite , or pre-trained word embeddings @xcite @xcite . even for neural topic models, there exists work on incorporating external knowledge, e.g., via word embeddings @xcite @xcite . in this paper, we show that adding contextual information to neural topic models provides a significant increase in topic coherence. this effect is even more remarkable given that we cannot embed long documents due to the sentence length limit in recent pre-trained language models architectures. concretely, we extend neural prodlda (product-of-experts lda) @xcite , a state-of-the-art topic model that implements black-box variational inference @xcite , to include contextualized representations. our approach leads to consistent and significant improvements in two standard metrics on topic coherence and produces competitive topic diversity results. contributions we propose a straightforward and easily implementable method that allows neural topic models to create coherent topics. we show that the use of contextualized document embeddings in neural topic models produces significantly more coherent topics. our results suggest that topic models benefit from latent contextual information, which is missing in bow representations. the resulting model addresses one of the most central issues in topic modeling. we release our implementation as a python library, available at the following link: https://github.com/milanlproc/ contextualized-topic-models . ## neural topic models with language model pre-training we introduce a combined topic model (com-binedtm) to investigate the incorporation of contextualized representations in topic models. our model is built around two main components: (i) the neural topic model prodlda @xcite and (ii) the sbert embedded representations @xcite . let us notice that our method is indeed agnostic about the choice of the topic model and the pre-trained representations, as long as the topic model extends an autoencoder and the pre-trained representations embed the documents. prodlda is a neural topic modeling approach based on the variational autoencoder (vae). the neural variational framework trains a neural inference network to directly map the bow document representation into a continuous latent representation. then, a decoder network reconstructs the bow by generating its words from the latent document representation 1 . the framework explicitly approximates the dirichlet prior using gaussian distributions, instead of using a gaussian prior like neural variational document models @xcite . moreover, prodlda replaces the multinomial distribution over individual words in lda with a product of experts @xcite . we extend this model with contextualized document embeddings from sbert (reimers and gurevych, 2019), 2 a recent extension of bert that allows the quick generation of sentence embeddings. this approach has one limitation. if a document is longer than sbert's sentence-length limit, the rest of the document will be lost. the document representations are projected through a hidden layer with the same dimensionality as the vocabulary size, concatenated with the bow repre1 for more details see @xcite ). figure 1 : high-level sketch of combinedtm. refer to @xcite for more details on the architecture we extend. ## experimental setting we provide detailed explanations of the experiments (e.g., runtimes) in the supplementary materials. to reach full replicability, we use open-source implementations of the algorithms. ## results we divide our results into two parts: we first describe the results for our quantitative evaluation, and we then explore the effect on the performance when we use two different contextualized representations. ## related work in recent years, neural topic models have gained increasing success and interest @xcite , due to their flexibility and scalability. several topic models use neural networks (larochelle and lauly, 2012; @xcite or neural variational inference @xcite @xcite @xcite . @xcite propose nvdm, an unsupervised generative model based on vaes, assuming a gaussian distribution over topics. building upon nvdm, dieng et al. ## conclusions we propose a straightforward and simple method to incorporate contextualized embeddings into topic models. the proposed model significantly improves the quality of the discovered topics. our results show that context information is a significant element to consider also for topic modeling.