Extracting Topics with Simultaneous Word Co-occurrence and Semantic Correlation Graphs: Neural Topic Modeling for Short Texts (2021)

Abstract: Short text nowadays has become a more fashionable form of text data, e.g., Twitter posts, news titles, and product reviews. Extracting semantic topics from short texts plays a significant role in a wide spectrum of NLP applications, and neural topic modeling is now a major tool to achieve it. Motivated by learning more coherent and semantic topics, in this paper we develop a novel neural topic model named Dual Word Graph Topic Model (DWGTM), which extracts topics from simultaneous word co-occurrence and semantic correlation graphs. To be specific, we learn word features from the global word co-occurrence graph, so as to ingest rich word co-occurrence information; we then generate text features with word features, and feed them into an encoder network to get topic proportions per-text; finally, we reconstruct texts and word co-occurrence graph with topical distributions and word features, respectively. Besides, to capture semantics of words, we also apply word features to reconstruct a word semantic correlation graph computed by pre-trained word embeddings. Upon those ideas, we formulate DWGTM in an auto-encoding paradigm and efficiently train it with the spirit of neural variational inference. Empirical results validate that DWGTM can generate more semantically coherent topics than baseline topic models.

## introduction the topic modeling family targets at learning latent topic representations from text document collections @xcite . during the past decades, it has been extensively applied in many tasks of natural language processing, e.g., sentiment analysis @xcite , summarization @xcite and classification @xcite , to name just a few. conventional topic models such as latent dirichlet allocation (lda) @xcite are often inferred by approximate inference methods, e.g., mean-field variational inference @xcite and collapsed gibbs sampling @xcite , which require model-specific derivations. the recent inference method with neural networks, such as variational auto-encoder (vae) @xcite , works in a black-box manner, providing a more generic and flexible solution to topic models beyond traditional approximate inference methods. broadly speaking, the models inferred with neural networks are referred to as neural topic models, and they have been recently drawn much more attention from the natural language processing community @xcite @xcite . unfortunately, whether for conventional or neural topic models, they tend to perform poorly on short text, a more fashionable and significant form of text data, e.g., twitter posts, news titles, and product reviews. the main reason is that short texts lack document-level word co-occurrences, known as the sparsity problem, which hinders models to capture coherent word patterns. many conventional topic models have been developed to handle short texts. for example, given very few words per-text, dirichlet multinomial mixture (dmm) @xcite constrains that each text covers a signal topic. biterm topic model (btm) @xcite directly learns topics from corpus-level word co-occurrence patterns. recently, there are also few attempts of neural topic models aiming to address the sparsity problem of short texts. graphbtm @xcite extracts topics from word graphs of randomly drawn mini-corpus. negative sampling and quantization topic model (nqtm) @xcite applies a topic distribution quantization method to pursue peakier topic proportions of texts. as reported in @xcite , those neu-ral topic models can empirically induce coherent topics from short texts. motivated by learning more coherent and semantic topics, in this paper we develop a novel neural topic model for short texts, namely dual word graph topic model (dwgtm). as the name suggests, in dwgtm we apply two word graphs, including the word co-occurrence graph constructed by aggregating word co-occurrence patterns of each text to alleviate the sparsity problem, and the word semantic correlation graph generated by using the pre-trained word embeddings to capture the semantic information of words. specifically, we formulate dwgtm in an auto-encoding paradigm with four main components: (1) we encode the word co-occurrence graph as word features by applying a graph convolutional network (gcn) module @xcite . (2) for each text, we construct its feature with corresponding word features, and encode it as topic proportion. in a nutshell, the major contributions of this paper are listed below: in this section, we briefly review related works on conventional topic modeling of short texts and neural topic modeling. ## the proposed dwgtm model in this section, we introduce the proposed dual word graph topic model (dwgtm). for convenience, the important notations used in this paper are summarized in ## experiment datasets. in the experiments, we select three benchmark short text datasets: trec, foot_1 googlenews, foot_2 and yahooanswer. 4 for all datasets, we remove digits and words with term frequencies less than 20. stop words and non-english words are filtered out by nltk. 5 for clarity, the statistics of those datasets are listed in baseline topic models. we select 8 existing baselines, including 4 conventional topic models and 4 neural topic models. following their original papers, the important implementation details of all baselines are described below. the model applies a 3-layer mlp encoder with 100 hidden neurons. for dwgtm, we apply a 2-layer gcn wcg-encoder and a 3-layer mlp tp-encoder, where the hidden neurons of both encoders are set as 100-300-400-300-k. to avoid posterior collapsing, we adopt 0.4 dropout, batch normalization, and a shallower 1-layer text-decoder. the threshold γ * is set to 0.6 for trec, and 0.8 for google-news and yahooanswer. scale parameters are set as λ 1 = 0.1, λ 2 = 0.1, λ 3 = 1. the number of epochs is 900 and mini-batch size is 200. to construct g s , we employ the 300-dimensional glove 14 embeddings trained on wikipedia2014 and giga-word5. for fair comparisons, the baselines requiring word embeddings use the same glove embeddings. evaluation metrics. to evaluate the topic quality, we adopt two metrics: topic coherence (tc) and topical semantics coherence (tsc). first, tc is the most popular topic quality metric that measures the co-occurrence statistics between top-m words of topics. here, we compute the tc scores with the public tc project of palmetto, 15 where, especially, the setting of c v is applied. second, we propose a novel metric named tsc to measure the semantic coherence of topics. analogy to tc, we suppose that higher similarities between top-m words of topics imply better semantic coherence for topics. accordingly, tsc can be defined as follows: