Combating Label Sparsity in Short Text Topic Modeling via Nearest Neighbor Augmentation (2024)

Abstract: Extracting semantic topics from short texts presents a significant challenge in the field of data mining. While efforts have been made to mitigate data sparsity issue, the limited length of short documents also results in the absence of semantically relevant words, causing biased evidence lower bound and incomplete labels for likelihood maximization. We refer to this issue as the label sparsity problem. To combat this problem, we propose kNNTM, a neural short text topic model that incorporates a k -Nearest-Neighbor-based label completion algorithm by augmenting the reconstruction label with k -nearest documents to complement these relevant but unobserved words. Furthermore, seeking a precise reflection of distances between documents, we propose a fused multi-view distances metric that takes both local word similarities and global topic semantics into consideration. Extensive experiments on multiple public short-text datasets show that kNNTM model outperforms the state-of-the-art baseline models and can derive both high-quality topics and document representations.

## introduction depiste the success of topic models in numerous nlp tasks @xcite for uncovering the underlying semantic concepts @xcite , traditional topic models often suffer from poor performances when applied to short text contents, e.g., social media posts and news * corresponding author headlines @xcite . this deficiency can be attributed to the lack of word co-occurrence information due to the limited length for a single short document, known as the data sparsity problem @xcite . many topic models have been developed to overcome the data sparsity issue. the dirichlet multinomial mixture (dmm) model @xcite @xcite constraints that each short text is generated by a single topic. biterm topic model @xcite utilizes the rich corpus-level word co-occurrence patterns for inferring topics. and some works @xcite @xcite aggregate semantically similar texts into long pseudodocuments. recently, with the developments of neural topic models (ntms) @xcite , there are also attempts to mitigate the data sparsity issue by utilizing biterm graph @xcite and topic quantization techniques @xcite . though these above works have achieved good performances and mitigated the data sparsity issue to some extent, there are still problems that they ignore. under the variational autoencoding framework, current mainstream ntms are optimized by the maximum likelihood objective, which is achieved by maximizing the evidence lower bound (elbo). however, the limited length of short texts results that only a few words get described in a document, while many other semantically relevant words remain unobserved @xcite . as shown in the motivating example in figure 1 , the document in the center talks about cpu chips and contains words like 'chip' and 'intel'. however, many semantically related words like 'core', 'processor', and 'cpu' remain uncovered. such 'incomplete' short documents will lead to a biased evidence lower bound as the possibilities of those unobserved but relevant words are completely ignored, resulting in a sub-optima optimization of the maximum likelihood. to be more specific, the absence of these relevant words leads to an incomplete target for the reconstruction objective during variational autoencoding, which makes the probabilities of the absent words get inappropriately suppressed and results in biased training signals. this problem, different from the data sparsity problem of the input data, is referred to as the label sparsity problem in this paper. inspired by the above observation, we propose to explicitly augment the reconstruction target in short text ntm with semantically related words to provide unbiased training signals. one direct approach to derive these words is to leverage the similarities between pre-trained word embeddings. however, word embeddings trained with general corpora may not capture the word co-occurrence patterns from a specific domain. moreover, simply relying on word similarities ignores the context information on the document level. in this paper, we propose knntm, a short text topic modeling framework, which incorporates a k-nearest-neighborbased label completion algorithm by aggregating k documents semantically closest to the target document to augment its reconstruction label. as illustrated in figure 1 , documents with semantically relevant words are retrieved by k-nearest neighbor searching, and the label distribution gets augmented by complementing the probabilities of originally unobserved words. the knn approach is shown to be effective in multiple fields for in-formation supplementation and data completion, like machine translation @xcite , healthcare prediction @xcite , and computer vision @xcite . in our scenario of short text topic modeling, aggregating k nearest documents for label completion helps to make full use of the word co-occurrence information and document relations from the original dataset. however, one remaining challenge of the knnbased label completion is to seek a proper distance metric that could precisely reflect the similarities between short documents with scarce context. a good metric should reflect both the word-level similarities and the global semantic resemblance. therefore, we propose a multi-view distance metric by fusing the distances from the input space and the semantic space to leverage both local and global similarities information. the distance metric in the input space depicts the local word similarity, which is defined with the optimal transport distance between bag-of-words distributions, with cost functions built upon word similarities from both general corpora and the specific dataset. and the metric in the hidden space reflects the global semantic resemblance, which is defined through the lens of topic semantics with the similarities between document-topic distributions. with the fused multiview distance metric, we can take various factors into account when evaluating the distances between documents, and provide a reasonable distance metric for the knn algorithm. our contributions are summarized as follows: • we identify the label sparsity problem in short text neural topic modeling, and propose a novel topic modeling framework, knntm, to combat this issue by a knn-based label completion algorithm with similar document aggregation. ## experimental results we evaluate the topic models from two perspectives: topic-word distribution and document-topic distribution. for the former, we assess the quality of topics based on coherence and diversity. regarding the latter, we utilize the performances from the clustering task as previous studies (zhao et al., 1 https://github.com/qiang2100/sttm 2021; @xcite . to verify models' effectiveness under different topic numbers, following previous work @xcite , we conducted experiments under 50 and 100 topics, respectively. ## conclusions in this paper, we identify the label sparsity problem in short text topic modeling, resulting from the inherently limited document length. subsequently, we design an novel neural short text topic modeling framework dubbed knntm, which mitigates the label sparsity problem with a knn label completion module that aggregates semantically similar documents to augment the reconstruction labels. to effectively find similar documents, a fused multiview distance metric is proposed considering both local word similarities and global document semantics. extensive experiments show that knntm outperforms the baselines and can generate highquality topic and document representations.