Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics (2022)

Abstract: Recent work incorporates pre-trained word embeddings such as BERT embeddings into Neural Topic Models (NTMs), generating highly coherent topics. However, with high-quality contextualized document representations, do we really need sophisticated neural models to obtain coherent and interpretable topics? In this paper, we conduct thorough experiments showing that directly clustering high-quality sentence embeddings with an appropriate word selecting method can generate more coherent and diverse topics than NTMs, achieving also higher efficiency and simplicity.

## introduction topic modelling is an unsupervised method to uncover latent semantic themes among documents @xcite . neural topic models (ntms) @xcite incorporating neural components have significantly advanced the modelling results than the traditional latent dirichlet allocation (lda; @xcite . later, contextualized word and sentence embeddings produced by pretrained language models such as bert @xcite have demonstrated the state-of-the-art results in multiple natural language processing (nlp) tasks @xcite , which attracts attentions from the topic modelling community. recent work has successfully incorporated these contextualized embeddings into ntms, showing improved topic coherence than conventional ntms that use bag-of-words (bow) as document representations @xcite . despite the promising performance, existing ntms are generally based on a variational autoencoder framework (vae; kingma and welling 2014), which suffers from hyper-parameters tuning and computational overheads @xcite . moreover, the integration of the pre-trained embeddings to the standard vae framework adds additional model complexity. with high-quality contextualized document representations, do we really need sophisticated ntms to obtain coherent and interpretable topics? recent work @xcite @xcite has shown that directly congregating contextualized embeddings can get semantically similar word or document clusters. specifically, @xcite cluster vocabulary-level word embeddings and obtain top words from each cluster using weighing and re-ranking, while @xcite consider polysemy and perform token-level clustering. however, the use of term frequency (tf) to select topic words fails to capture the semantics of clusters precisely because words with high frequency may be common across different clusters. grootendorst (2020) propose a class-based term frequency-inverse document frequenc (c-tf-idf) method that extract important words from each clustered documents, which tends to choose representative words within each cluster to form topics. however, it overlooks the global semantics between clusters which could be incorporated. in addition, all above works only compare the performance with the traditional lda while ignoring the promising ntms proposed recently. the performance of the clustering-based topic models is still yet uncovered. is neural topic modelling better than simple embedding clustering? this work compares the performance of ntms and contextualized embeddingbased clustering systematically. our main focus is to provide insights by comparing the two paradigms for topic models, which has not been investigated before. we employ a straightforward framework for clustering. in addition, we explore different strategies to select topic words for clusters. we evaluate our approach on three datasets with various text lengths. our contributions are as follows: first, we find that directly clustering high-quality sentence embeddings can generate as good topics as ntms, providing a simple and efficient solution to uncover latent topics among documents. second, we propose a new topic word selecting method, which is the key to producing highly coherent and diverse topics. third, we show that the clustering-based model is robust to the length of documents and the number of topics. reducing the embedding dimensionality negligibly affects the performance but saves runtime. from our best knowledge, we are the first to compare with ntms, using contextualized embeddings that produced by various transformer-based models. ## models this study compares embedding clustering-based models with lda and a series of existing ntms as follows. implementation details are supplied in appendix a. lda @xcite : the representative traditional topic model in history that generates topics via document-topics and topic-words distributions. prodlda @xcite : a prominent ntm that employs the vae (kingma and welling, 2014) to reconstruct the bow representation. combinedtm @xcite : extends prodlda by concatenating the contextualized sbert @xcite embeddings with the original bow as the new input to feed into the vae framework. zeroshottm @xcite : also builds upon prodlda, but it replaces the original bow with sbert embeddings entirely. bert+km @xcite : a clustering-based method that first uses k-means (km) to cluster word embeddings, then apply tf to weight and re-rank words to obtain topic words. bert+umap+hdbscan (i.e., bertopic) (grootendorst, 2020): a clustering-based method that first leverages @xcite to cluster bert embeddings of the sentences and uniform manifold approximation projection (umap) @xcite to reduce embedding dimensions, then use a class-based tfidf (i.e. c-tf-idf) to select topic words within each cluster. note that bertopic may not generate the specified number of topics. contextual embeddings+umap+km (our method cetopic): we use a simple clustering framework with contextualized embeddings for topic modelling, as shown in figure 1 . we first encode pre-processed documents to obtain contextualized sentence embeddings through pre-trained language models. after that, we lower the dimension of the embeddings before applying clustering methods (e.g., k-means; km) to group similar documents. each cluster will be regarded as a topic. finally, we adopt a weighting method to select representative words as topics. we believe that high-quality document embeddings are critical for clustering-based topic modelling. we thus experiment with different embeddings including bert, roberta @xcite , and sbert. we also adopt simcse @xcite , a recently proposed sentence embeddings of contrastive learning, that has shown the state-ofthe-art performance on multiple semantic textual similarity tasks. both supervised and unsupervised simcse are investigated in our experiment (e.g., pre-trained contextualized sentence embeddings often have high dimensionalities. to reduce the computational cost, we apply the umap in our implementation to reduce the dimensionality while maintaining the essential information of the embeddings. we find that reducing dimensionality before clustering has a negligible impact on performance (section 4.4). we cluster the dimension-reduced sentence embeddings using k-means because of its efficiency and simplicity. semantically close documents are gathered together, and each cluster is supposed to represent a topic. tify semantics of topics. inspired by previous works @xcite ; grootendorst, 2020), we explore several weighting metrics to obtain topic words in clusters. let n t,d be the frequency of word t in document d, t ′ n t ′ ,d be the total words' frequency in the document, and d be the entire corpus. term frequency-inverse document frequency (tfidf) is defined as @xmath0,d t ′ n t ′ ,d besides the two local cluster-based strategies, we further incorporate the global word importance with local term frequency within each cluster: ## conclusion we conduct a thorough empirical study to show that a clustering-based method can generate commendable topics as long as high-quality contextualized sentence embeddings are used, together with an appropriate topic word selecting strategy. compared to neural topic models, clustering-based models are more simple, efficient and robust to various document lengths and topic numbers, which can be applied in some situations as an alternative.