G lo COM : A Short Text Neural Topic Model via Global Clustering Context (2025)

Abstract: Uncovering hidden topics from short texts is challenging for traditional and neural models due to data sparsity, which limits word co-occurrence patterns, and label sparsity, stemming from incomplete reconstruction targets. Although data aggregation offers a potential solution, existing neural topic models often overlook it due to time complexity, poor aggregation quality, and difficulty in inferring topic proportions for individual documents. In this paper, we propose a novel model, **GloCOM** (**Glo**bal **C**lustering C**O**ntexts for Topic **M**odels), which addresses these challenges by constructing aggregated global clustering contexts for short documents, leveraging text embeddings from pre-trained language models. GloCOM can infer both global topic distributions for clustering contexts and local distributions for individual short texts. Additionally, the model incorporates these global contexts to augment the reconstruction loss, effectively handling the label sparsity issue. Extensive experiments on short text datasets show that our approach outperforms other state-of-the-art models in both topic quality and document representations.

## introduction topic models @xcite have proven effective in discovering topics within a corpus and providing a high-level representation of documents. topic models are applied in various domains, including text mining @xcite , bioinformatics @xcite , and recommender systems @xcite and streaming learning @xcite @xcite @xcite . however, while they perform well with long texts, these models often struggle with short data @xcite @xcite . datasets containing short documents, such as headlines, comments, or search snippets, offer limited information on word co-occurrence @xcite , essential for identifying latent topics. this challenge, known as data sparsity, significantly hinders the ability of recent models to generate high-quality topics. moreover, the brevity of short texts introduces label sparsity @xcite , where unobserved but relevant words are ignored in the evidence lower bound, causing biased reconstruction loss in variational autoencoder (vae)-based neural topic models @xcite . document aggregation has effectively addressed short text topic modeling challenges @xcite @xcite . however, modern neural network-based topic models @xcite @xcite have not paid much attention to this approach due to the limitations demonstrated in traditional research. particularly, aggregation approaches that rely on auxiliary information @xcite are often restricted to specific data types, while self-aggregation methods face challenges such as high time complexity or overfitting as data volume increases @xcite . moreover, some methods are unable to infer topic distributions for individual documents @xcite . the knntm @xcite model is the first short text neural topic model to address label sparsity using knn-based document aggregation. by enhancing the reconstruction target with semantically related documents, the model can leverage word co-occurrence patterns and the relationships between documents in the dataset. although this approach has proven effective, knntm still faces significant time costs due to optimal transport measures between every pair of documents in the cor-cluster 1: #1: nokia lumia launch #2: moto order start shipping december support #3: microsoft officially launched xbox console gamers cluster 2: #4: black friday sale cyber monday #5: local shop start sale thanksgiving day #6: shopper black friday to address these issues, we propose a novel topic model called glocom, which constructs a neural topic model using global clustering context. our model first clusters the document set and then creates global contexts (or global documents) by merging short documents (or local documents) within each cluster. the topic model then incorporates global and local texts, yielding global and local topic distributions. each local topic distribution is derived from the global distribution of its corresponding context, supplemented by a learnable noise parameter. this approach enables the model to harness the benefits of document aggregation to enhance word co-occurrences and handle the data sparsity issue while inferring representations for individual documents. furthermore, advancements in foundation models @xcite have improved short text processing effectiveness for downstream tasks. consequently, we employ pre-trained language model (plm) embeddings @xcite to represent texts during the clustering process. as shown in besides, we augment the model's reconstruction targets by combining the original short text with the globally aggregated documents mentioned above. this approach allows us to globally capture absent yet relevant words related to the original input, such as "shop" and "shopper" in documents #5 and #6, which belong to the same cluster, as shown in ## methodology we propose a novel topic model framework for short texts, which introduces an effective clusterbased text aggregation method and an innovative inference for both the global enhanced context and each individual document. furthermore, we reuse the global context to augment the vae's reconstruction labels, addressing the label sparsity issue. ## conclusion in this paper, we propose a novel neural model for short text topic modeling, called glocom, which leverages aggregated global clustering context and integrates this context into the reconstruction target. our experiments demonstrate that our method effectively addresses both data and label sparsity challenges, outperforming state-of-the-art models and producing high-quality topics and document representations for short texts.