Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder (2020)

Abstract: Topic models have been prevailing for many years on discovering latent semantics while modeling long documents. However, for short texts they generally suffer from data sparsity because of extremely limited word co-occurrences; thus tend to yield repetitive or trivial topics with low quality. In this paper, to address this issue, we propose a novel neural topic model in the framework of autoencoding with a new topic distribution quantization approach generating peakier distributions that are more appropriate for modeling short texts. Besides the encoding, to tackle this issue in terms of decoding, we further propose a novel negative sampling decoder learning from negative samples to avoid yielding repetitive topics. We observe that our model can highly improve short text topic modeling performance. Through extensive experiments on real-world datasets, we demonstrate our model can outperform both strong traditional and neural baselines under extreme data sparsity scenes, producing high-quality topics.

## introduction in addition to formal documents, short texts play an increasingly more important role in the era of information explosion where people could instantly share ideas, feelings, and comments via short text fragments, including tweets, headlines, and product reviews, etc. the latent semantics or topics discovered among these short texts can be utilized in many applications, such as content summarization @xcite , classification @xcite , and recommendations @xcite . however, conventional topic models @xcite work reasonably well on various kinds of long documents, but perform poorly on short texts. the main underlying reason is that the co-occurrence information from short texts is extremely limited as known as the data sparsity sports scores games soccer league tennis ncaa players football sports tennis soccer hockey games football beach match players sports match cup hockey olympic football players sport league sports football sport league games tennis champions club sports football league game tennis players hockey games scores bad additional abstract aspectj behave displayed customise accept abstract behave accept additional bad displayed customise abstract accept behave additional adding long many administration in this paper, we aim to design a model that can generate high-quality topics from short texts and is more robust to rigorous data sparsity scenarios without any auxiliary corpus. different from previous methods, we propose a new negative sampling and quantization topic model (nqtm) in an auto-encoding framework to address the unsupervised short text modeling problem including two essential and novel methods. first, for short texts, we need peakier topic distributions for decoding since short texts cover few primary topics, like dirichlet multinomial mixture (dmm) @xcite that assumes each short text only covers one topic. in the autoencoding framework, a possible and straightforward way is using gumbel-softmax @xcite , but its performance is highly determined by the temperature parameter that necessarily needs to be tuned across topic numbers and corpora; therefore, it may not guarantee high-quality topics. another way is quantizing the latent representations like vq-vae (van den oord and vinyals, 2017). unfortunately, the original quantization of vq-vae is for image generation and cannot produce peakier distributions for short text topic modeling. therefore, we propose the novel topic distribution quantization for short texts by separably mapping topic distributions into an appropriate defined embedding space. with this new method, our model can naturally encourage discretization to flexibly yield peakier distributions for decoding, resulting in much better topic quality performance. second, we propose a new negative sampling decoder to improve the topic diversity performance. as mentioned previously, short texts are extremely sparse inputs, so the learning signals are too weak to converge to a good local minimum, notably in an unsupervised learning fashion, leading to repetitive topics. therefore, instead of using a straightforward log-likelihood objective, we propose a negative sampling decoder with the reconstruction by selecting target words from assigned topics and negative words from the topics that are unlikely to be assigned. it acts as an inductive bias that encourages the topic-word distributions to be pushed away from each other, resulting in a better learning objective for generating diverse topics. the main contributions foot_0 of this paper can be concluded as neural topic models more recently, deep neural networks have shown great potential for learning complicated distributions for unsupervised models. due to the success of variational autoencoder (vae) @xcite , various neural topic models are proposed @xcite . neural variational document model (nvdm) @xcite is the first vae-based neural topic model that adopts the reparameterization trick of gaussian distributions and achieves remarkable results on normal text topic modeling. some extensions like gaussian softmax construction (gsm) have been explored in @xcite . product of expert lda (prodlda) is proposed by @xcite using logistic normal distribution due to the difficulty of taking the reparameterization trick for dirichlet distribution, which is important for topic modeling. topic memory network (tmn) @xcite is proposed for supervised short text topic modeling and classification with pre-trained word embeddings, incorporating the neural topic model @xcite with memory networks @xcite . different from these neural topic models, the proposed model aims to improve short text topic modeling without any extra information. our model relies on the novel topic distribution quantization to discrete the latent representations in the auto-encoding framework instead of the vae assumption. meanwhile, a new objective under the negative sampling decoder replaces the traditional log-likelihood maximization objective to especially alleviate the data sparsity of short texts. ## conclusion in this paper, for short text topic modeling, we propose the negative sampling and quantization topic model (nqtm) with a novel topic distribution quantization mechanism to yield peakier distributions and a new negative sampling decoder to enrich the learning signals. experiments on benchmark datasets quantitatively and qualitatively show our model significantly outperforms baselines to overcome the data sparsity problem of short texts. future works could focus on employing the proposed model in more downstream tasks.