Topic Modeling for Short Texts via Optimal Transport-Based Clustering (2025)

Abstract: Discovering topics and learning document representations in topic space are two crucial aspects of topic modeling, particularly in the short-text setting, where inferring topic proportions for individual documents is highly challenging. Despite significant progress in neural topic modeling, effectively distinguishing document representations as well as topic embeddings remains an open problem. In this paper, we propose a novel method called **En**hancing Global **C**lustering with **O**ptimal **T**ransport in Topic Modeling (EnCOT). Our approach utilizes an abstract global clusters concept to capture global information and then employs the Optimal Transport framework to align document representations in the topic space with global clusters, while also aligning global clusters with topics. This dual alignment not only enhances the separation of documents in the topic space but also facilitates learning of latent topics. Through extensive experiments, we demonstrate that our method outperforms state-of-the-art techniques in short-text topic modeling across commonly used metrics.

## introduction topic models aim to uncover a set of latent topics from a document collection by analyzing word cooccurrence patterns. each topic represents a coherent semantic concept and is characterized by related words. additionally, topic models estimate the topic distribution within each document (topic proportions), shedding light on their underlying meanings. traditional approaches to topic modeling rely on either probabilistic graphical models @xcite @xcite or non-negative matrix factorization techniques @xcite . recently, neural topic models (ntms) have emerged as a powerful alternative, leveraging advances in deep learning. unlike conventional methods, ntms utilize deep neural networks to model distributions, allowing efficient and flexible parameter inference through automatic gradient back-propagation, as exemplified in @xcite . this adaptability enables researchers to customize model architectures to suit a wide range of application scenarios. conventional topic models often face significant challenges when applied to short texts. the primary issue lies in their word co-occurrence information to infer latent topics. in short texts, this information is highly sparse due to the limited context available, making it difficult for topic models to extract meaningful patterns @xcite @xcite . this issue, commonly referred to as data sparsity @xcite , hampers the models' ability to generate high-quality topics and has therefore become a focal point of interest within the research community. numerous studies have been proposed to tackle the issue of data sparsity in short texts. @xcite develop nqtm, which applies vector quantization to doc-topic distributions based on the concept from @xcite . @xcite propose leveraging word co-occurrence and semantic correlation graphs to enhance the learning signals for short texts. @xcite integrate entity vector representations into a neural topic model (ntm) for short texts, learning these vectors from manually curated knowledge graphs. building upon nqtm, @xcite introduce tsctm, a contrastive learning method designed to capture topic semantics more effectively by modeling similarity relationships among short texts. a significant limitation of previous studies is their inability to disentangle doc-topic representations, leading to poor performance in downstream tasks such as document clustering, document classification. without clear separation in these representations, it becomes challenging to identify meaningful groupings of documents. this drawback underscores the need for methods that enhance the interpretability and structure of doc-topic distributions to better support tasks requiring highquality clustering and classification. to combat the challenges of data sparsity and improve topic coherence, @xcite introduce glo-com, a neural topic model that integrates global clustering context. the model begins with clustering documents and then creating global contexts by merging short texts within each cluster, thereby enhancing word co-occurrence statistics. additionally, glocom leverages pre-trained language model (plm) embeddings for clustering, which significantly improves the semantic of document clusters and the quality of inferred topics. as demonstrated with nmi and purity score, this plm-enhanced global clustering contributes to superior topic coherence and topic diversity, highlighting the model's effectiveness in semantic richness and clustering precision. despite its innovations, glocom and existing neuron topic modeling approaches face two significant challenges. first, none of these methods directly disentangle document representations by explicitly designing loss functions to achieve this. while glocom is the first model to leverage clustering for document separation, its improved performance primarily relies on clustering and data aggregation techniques rather than a targeted disentangling strategy. second, like document separation, this model lacks explicit regularization on topic representations. in glocom, the loss function does not directly account for enhancing topic quality. this highlights the need for a method that actively separates topics, ensuring distinct and coherent topic representations. building on these two findings, we propose an approach to tackle the identified challenges. clustering serves as an intuitive and effective method for data point separation by naturally grouping similar data points within a cluster while segregating dissimilar ones into distinct clusters. this approach aligns closely with the concept of proximity in feature space, where similar entities are grouped together due to shared attributes. with respect to document representation in topic space, clustering ensures that closely related documents within the same topic are grouped, while unre-lated documents are assigned to separate clusters. similarly, topics can be hierarchically organized, with broader themes encompassing more specific subtopics. for instance, "physics" and "biology" might collectively form the higher-level topic "science," while "politics," "sports," and "technology" can be grouped under "news." this hierarchical arrangement offers a more natural and accurate representation of the data, capturing both broad and nuanced relationships within the dataset. for choosing a clustering method, we rely on optimal transport (ot). ot treats documents and topics as distributions and computes the minimal "transport cost" required to map them to cluster centers (centroids) -a novel abstract global clusters concept. this approach allows ot to dynamically adapt to the semantic relationships within the data, ensuring that documents with similar content are grouped together towards a same centroid and topics with overlapping themes are aligned to a share centroid. by aligning documents to centroids, ot enhances document representation, leading to better separation and coherence. likewise, aligning topics with centroids ensures that topics are enriched with higher-level thematic structures, allowing for a hierarchical understanding of the corpus. this dual application of ot not only enhances the separation of documents but also improves the coherence and diversity of topics. the contributions of this paper are summarized as follows: ## preliminaries in this section, we provide an overview of topic modeling, including the problem settings and notations. we then introduce the background of a state-of-the-art method -global clustering contexts for topic models (glocom) @xcite , which serves as the foundation for illustrating how our approach can be applied to various models. ## methodology figure 2 illustrates the architecture of the glocom integrated with our novel encot. in the glocom framework, short texts are first clustered using embeddings derived from a pre-trained language model (plm) and global documents are created by concatenating texts within each cluster. the short texts are processed through a multi-layer perceptron (mlp) to generate an adaptive variable. in parallel, the global documents are transformed through another mlp to produce a global distribution. the document representation is obtained as the dot product of the adaptive variable and the global distribution, integrating information from the global clusters. word embeddings (w) and topic embeddings (t ) are employed to construct topics β following the methodology described in @xcite . the augmented documents are central to compute the reconstruction loss, serving as essential components within the glocom framework @xcite . the newly introduced elements-encot with two ot losses-are highlighted in green ellipses. as depicted in the figure, the ot losses are applied directly to the representations of topic embeddings and document-topic vectors, ensuring that these representations remain distinct, as discussed in the previous analysis. ## conclusion in this paper, we introduce a novel approach called enhancing global clustering with optimal transport in topic modeling (encot), designed to simultaneously address the challenges of document separation and topic separation. encot leverages clustering concepts within the optimal transport framework to achieve this distinction effectively. comprehensive experiments validate the effectiveness of encot, demonstrating its ability to achieve robust neural topic modeling by ensuring clear separation between topics and between documents.