Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning (2022)

Abstract: To overcome the data sparsity issue in short text topic modeling, existing methods commonly rely on data augmentation or the data characteristic of short texts to introduce more word co-occurrence information. However, most of them do not make full use of the augmented data or the data characteristic: they insufficiently learn the relations among samples in data, leading to dissimilar topic distributions of semantically similar text pairs. To better address data sparsity, in this paper we propose a novel short text topic modeling framework, Topic-Semantic Contrastive Topic Model (TSCTM). To sufficiently model the relations among samples, we employ a new contrastive learning method with efficient positive and negative sampling strategies based on topic semantics. This contrastive learning method refines the representations, enriches the learning signals, and thus mitigates the sparsity issue. Extensive experimental results show that our TSCTM outperforms state-of-the-art baselines regardless of the data augmentation availability, producing high-quality topics and topic distributions.

## introduction topic models aim to discover the latent topics of a document collection and infer the topic distribution of each document in an unsupervised fashion @xcite . due to the effectiveness and interpretability, topic models have been popular for decades with various downstream applications @xcite @xcite . however, despite the success on long texts, current topic models generally cannot handle well short texts, such as tweets, headlines, and comments @xcite . the reason lies in that topic models rely on word co-occurrence information to infer latent topics, but such information is 1 our code is available at https://github.com/ bobxwu/tsctm . to overcome the data sparsity issue, traditional wisdom can be mainly categorized into two lines: (i) augment datasets with more short texts containing similar semantics @xcite @xcite . this way can feed extra word co-occurrence information to topic models. (ii) due to the limited context, many short texts in the same collection, such as tweets from twitter, tend to be relevant, sharing similar topic semantics @xcite ; to leverage this data characteristic, models such as dmm @xcite and state-of-the-art nqtm @xcite learn similar topic distributions from relevant samples. these two lines of thought have been shown to achieve good performance and mitigate data sparsity to some extent. however, existing short text topic models neither make full use of the augmented data nor the crucial data characteristic. to begin with, an augmented text is expected to have a similar topic distribution as the original text since they share similar topic semantics, but existing approaches tend to overlook this important relation between samples. as shown in figure 1b , text x (i) and its augmented view x (i) + have similar topic semantics, but their topic distributions inferred by nqtm are far from similar. moreover, guided by the aforementioned data characteristic, state-of-the-art methods like nqtm attempt to learn similar topic distributions for relevant samples, yet they could inappropriately do so. figure 1b shows that text x (i) and x (ℓ) are relevant, but their learned topic distributions are dissimilar; x (i) and x (j) are irrelevant, but theirs are similar. in a word, current approaches insufficiently model the relations among samples in data, which hinders fully addressing the data sparsity issue. to better mitigate data sparsity, we in this paper propose topic-semantic contrastive topic model (tsctm), a novel short text topic modeling framework that unifies both cases with and without data augmentation. to be specific, tsctm makes full use of relations among samples with a novel topicsemantic contrastive learning method. in the case without data augmentation, tsctm effectively samples positive and negative text pairs based on topic semantics. in the case with data augmentation, tsctm also smoothly incorporates the relations between augmented and original samples, enabling better utilization of data augmentation. through the novel contrastive learning method, tsctm sufficiently models the relations among samples, which enriches the learning signals, refines the learning of representations, and thus mitigates the data sparsity issue (see figure 1c for an illustration). we summarized the main contributions of this paper as follows: they use gibbs sampling (griffiths and steyvers, 2004) or variational inference (blei et al., 2017) to infer model parameters. later, due to the effectiveness and brevity of variational autoencoder (vae, kingma and welling, 2014; rezende @xcite , many neural topic models have been introduced @xcite @xcite @xcite @xcite @xcite . among those methods, the most related one to this paper is nqtm @xcite . although nqtm also uses vector quantization to aggregate the short texts with similar topics, however, we note that our method differs significantly in that: (i) our tsctm framework uses the novel topic-semantic contrastive learning method that fully considers the relations among samples with effective positive and negative sampling strategies, while nqtm only considers the relations between samples with similar semantics. (ii) our tsctm framework can adapt to the case with data augmentation by sufficiently modeling the relations brought by augmented samples, achieving higher performance gains, while nqtm cannot fully incorporate such relations. contrastive learning the idea of contrastive learning is to measure the similarity relations of sample pairs in a representation space @xcite @xcite @xcite @xcite . it has been widely explored in the visual field, such as image classification @xcite , objective detection @xcite , and image segmentation @xcite . for text data, some studies use contrastive loss @xcite by sampling salient words from texts to build positive samples, but they could be inappropriate for short text topic modeling due to the limited context of short texts (shown in sec. 5.1). in contrast, our new framework can discover effective samples for learning contrastively based on the topic semantics and can smoothly adapt to the case with augmentations, both of which better fit the short text modeling context. ## methodology in this section, we first review the background of topic modeling. then we introduce topic-semantic contrastive learning, a novel approach for short text topic modeling. finally, we put this contrastive learning into the topic modeling context and propose our topic-semantic contrastive topic model. ## experimental setting in this section, we conduct comprehensive experiments to show the effectiveness of our method. ## conclusion in this paper, we propose tsctm, a novel and unified method for topic modeling of short texts. our method with the novel topic-semantic contrastive learning can refine the learning of representations through sufficiently modeling the relations among texts, regardless of the data augmentation availability. experiments show our model effectively alleviates the data sparsity issue and consistently outperforms state-of-the-art baselines, generating high-quality topics and deriving useful topic distributions of short texts.