CAST : Corpus-Aware Self-similarity Enhanced Topic modelling (2025)

Abstract: Topic modelling is a pivotal unsupervised machine learning technique for extracting valuable insights from large document collections. Existing neural topic modelling methods often encode contextual information of documents, while ignoring contextual details of candidate centroid words, leading to the inaccurate selection of topic words due to the *contextualization gap*. In parallel, it is found that functional words are frequently selected over topical words. To address these limitations, we introduce **CAST**: **C**orpus-**A**ware **S**elf-similarity Enhanced **T**opic modelling, a novel topic modelling method that builds upon candidate centroid word embeddings contextualized on the dataset, and a novel self-similarity-based method to filter out less meaningful tokens. Inspired by findings in contrastive learning that self-similarities of functional token embeddings in different contexts are much lower than topical tokens, we find self-similarity to be an effective metric to prevent functional words from acting as candidate topic words. Our approach significantly enhances the coherence and diversity of generated topics, as well as the topic model’s ability to handle noisy data. Experiments on news benchmark datasets and one Twitter dataset demonstrate the method’s superiority in generating coherent, diverse topics, and handling noisy data, outperforming strong baselines.

## introduction topic modelling is a powerful tool for extracting valuable insights and detecting topics of interest within a large collection of documents, where the conventional statistical-based models @xcite @xcite were typically developed based on the dominant latent dirichlet allocation (lda) model @xcite . however, lda relies on the bag-of-words (bow) model which has inherent limitations such as ignoring the semantic meaning of words, word order, and contextual information. to address these issues, recent studies have incorporated embedding techniques, such as word2vec @xcite , bert @xcite and sentence-bert @xcite , into topic modelling. for instance, @xcite integrates clustering algorithms with various embedding models to identify topics, demonstrating that this method achieves comparable performance to traditional lda while reducing complexity and runtime. similarly, top2vec @xcite and bertopic @xcite use embedding models to encode and cluster document embeddings based on their semantic similarities, representing each cluster as a topic. these models have shown significant advancements over lda in previous studies in terms of generating coherent and diverse topics that capture semantic relationships and context in the documents @xcite @xcite . however, a common limitation of the aforementioned models is their failure to fully consider contextual information within documents. although top2vec utilizes contextual embedding models to generate document embeddings, when encoding candidate topic words, it encodes them as standalone words, disregarding surrounding context. this has prevented taking into account the contextualized meanings of these words in the corpus of diverse topics. for instance, the embeddings for "bank" in the context of a financial institution and a riverbank would be identical, depending solely on the embedding model they use. this can lead to inaccurate topic word assignments, and failure to properly represent the topics, especially when documents have a domain shift compared to the training corpus of the embedding model. similarly, figure 1 : two modules to identify meaningful candidate topic words. module 1: word embeddings contextualized on the dataset. module 2: self-similarity scores to filter out functional words. purple points represent documents, with semantically similar ones clustered together. words with higher self-similarity scores (green) are selected over those with lower scores and assigned to their closest cluster centroid (topic vector: yellow point) as topic words (green points), rather than relying on general-domain topic words (red points). bertopic derives its topic representations from a bow approach which fails to leverage the advancements of embedding techniques. this method may have limitations such as neglecting the contextual information captured by the language model, which can result in missing crucial semantic nuances and word relationships within topics. another area where current topic modelling methods fall short lies in building meaningful topic words. topic models like lda and bertopic may inadvertently prioritize common terms and sometimes select functional words, such as stopwords, over topical words, despite extensive preprocessing. standard stopword lists, like those from nltk, may not fully capture all irrelevant terms for a specific corpus and often require prior knowledge to build a customized stopword list tailored to the dataset. to address the above limitations, we introduce cast: corpus-aware self-similarity enhanced topic modelling, a novel topic model that extends top2vec by introducing two novel modules: 1) word embeddings contextualized on the dataset and 2) self-similarity scores to filter out functional words and construct meaningful candidate topic words, as shown in figure 1 . more specifically, we first leverage embedding models to joint encode word and document embeddings. candidate topic words are encoded based on their usage within the documents, instead of statically encoded in the general domain as standalone tokens (the red point). second, we reduce the dimensionality of the document embeddings and cluster the reduced embeddings to create semantically similar topic clusters (purple points). third, inspired by the findings in contrastive learning that functional tokens have lower self-similarity scores than that of semantic tokens (xiao et al., 2023b), we employ a self-similarity threshold to effectively filter out functional words or less relevant words and keep meaningful candidate topic words. finally, we assign the candidate words closest to their cluster centroid (yellow point) as the corresponding topic words (green points). the contributions of this paper are three-fold: ## results the main results are shown in in terms of tc, cast models exhibit consistently high tc scores on both traditional npmi and llm-based coherence metrics across all datasets. among the two variants, cast-mpnet achieves the highest npmi and llm-tc scores on the 20newsgroups and elon musk datasets. cast-minilm demonstrates consistent and competitive coherence on all datasets, with the best llm-tc score on bbc news, and the second-best on 20newsgroups. in terms of td, topclus stands out in td across all datasets. cast shows competitive performance compared to topclus, with the best llm-td on 20newsgroups, indicating a balance between topic coherence and diversity. the vanilla cast models (marked with *), which ablate the self-similarity module, still outperform baseline models in most cases, underscoring the efficacy of the contextualized embedding module to improve the model performance. however, compared with the full cast models, the vanilla models exhibit marginally lower coherence scores while maintaining robust diversity metrics. this suggests that the self-similarity module, by filtering out functional words, could enhance coherence at a slight cost to topic word diversity. we further explain this in section 5. qualitative results. in terms of lda and bertopic, they frequently generate noisy and semantically irrelevant words despite their limited meaningfulness. this is potentially because these two models do not leverage the semantic richness offered by word embeddings to identify topic words. topclus and bertopic models exhibit significant gaps in topic coverage. specifically, bertopic struggles to distinguish between topics 2 and 4 in the 20newsgroups dataset, leading to a conflation of topic words. this indicates a limitation in the model's ability to maintain distinct topic boundaries. topclus tends to cluster meaningless adjectives and verbs together as a topic. for example, "good", "long", "big", "small", "large" in ## ablation analysis in this section, we ablate the self-similarity thresholds to evaluate how varying thresholds affect tc and td. we tested cast-mpnet on the 20news-groups and bbc news datasets. as shown in figure 2 , increasing the selfsimilarity threshold initially enhances the model's performance, but experiences a marked decline after a higher threshold. as tested on the 20news-groups and bbc news datasets, tc and llm-tc remain competitive and peak around a threshold of 0.4 in most cases, while td and llm-td remain stable with minor fluctuations up to this point. however, beyond a self-similarity score of 0.5, the coherence and diversity metrics experience a significant decline. this pattern indicates that an optimal range of self-similarity scores can increase topic coherence and diversity, but excessively high thresholds may filter out meaningful words, thereby reducing the performance. in practical applications, determining the appropriate self-similarity threshold is crucial and will vary depending on the corpus and the embedding model used. in our experiment, we sorted and displayed the self-similarity scores of words across the corpus, and then manually selected a threshold that can effectively exclude most functional words while retaining the most meaningful ones. as shown in additionally, when setting self-similarity to 0, ablating the use of the self-similarity module, cast still outperforms other models under the same conditions in most cases (see the vanilla models in our approach uses similar methods as top2vec and bertopic for dimensionality reduction and embedding clustering. however, it differs in attaining the candidate topic word embeddings by averaging the embeddings of the word in different contexts across the corpus, to capture contextual information of the corpus. in contrast, top2vec typically extracts static word embeddings from pretrained embeddings. bertopic, instead of building word embeddings and finding candidate words whose embeddings are close to the topic embeddings, identifies topic words through a clusteringbased tf-idf approach. furthermore, our method introduces self-similarity scores as a novel mechanism for filtering candidate topic words. typically, functional words exhibit lower self-similarity scores in their embeddings compared to semantically meaningful words. by establishing an appropriate threshold, we can effectively exclude most functional words while retaining semantically significant terms. this could enhance topic coherence and improve interpretability for end-users. ## conclusion we propose an alternative topic modelling approach that extends top2vec and bertopic by using two modules: 1) word embeddings contextualized on the dataset and 2) self-similarity scores to filter out functional words and construct meaningful candidate topic words. these two modules are straightforward and flexible, making them adaptable components for other topic models. furthermore, they are independent of dimension reduction and document clustering which enhances cast's robustness against parameter changes in umap and hdbscan algorithms. moreover, these modules could be further enhanced by integrating new embedding models and advanced clustering techniques, thus offering a scalable framework for future developments in topic modelling. we evaluated cast on two benchmark datasets and one twitter dataset, demonstrating superior performance compared to the strong baseline models in identifying coherent and diverse topics.