Neural Topic Modeling by Incorporating Document Relationship Graph (2020)

Abstract: Graph Neural Networks (GNNs) that capture the relationships between graph nodes via message passing have been a hot research direction in the natural language processing community. In this paper, we propose Graph Topic Model (GTM), a GNN based neural topic model that represents a corpus as a document relationship graph. Documents and words in the corpus become nodes in the graph and are connected based on document-word co-occurrences. By introducing the graph structure, the relationships between documents are established through their shared words and thus the topical representation of a document is enriched by aggregating information from its neighboring nodes using graph convolution. Extensive experiments on three datasets were conducted and the results demonstrate the effectiveness of the proposed approach.

## introduction probabilistic topic models @xcite are tools for discovering main themes from large corpora. the popular latent dirichlet allocation (lda) @xcite and its variants @xcite @xcite are effective in extracting coherent topics in an interpretable manner, but usually at the cost of designing sophisticated and model-specific learning algorithm. recently, neural topic modeling that utilizes neuralnetwork-based black-box inference has been the main research direction in this field. notably, nvdm @xcite employs variational autoencoder (vae) @xcite to model topic inference and document generation. specifically, nvdm consists of an encoder inferring topics from documents and a decoder generating documents from topics, where the latent topics are constrained by a gaussian prior. @xcite argued that dirichlet distribution is a more appropriate prior for topic modeling than gaussian in nvdm and proposed prodlda that approximates the dirichlet prior with logistic normal. there are also attempts that directly enforced a dirichlet prior on the document topics. w-lda @xcite models topics in the wasserstein autoencoders @xcite framework and achieves distribution matching by minimizing their maximum mean discrepancy (mmd) @xcite , while adversarial topic model @xcite ) directly generates documents from the dirichlet prior and such a process is adversarially trained with a discriminator under the framework of generative adversarial network (gan) @xcite . recently, due to the effectiveness of graph neural networks (gnns) @xcite @xcite in embedding graph structures, there is a surge of interests of applying gnn to natural language processing tasks @xcite @xcite . for example, graphbtm @xcite is a neural topic model that incorporates the graph representation of a document to capture biterm cooccurrences in the document. to construct the graph, a sliding window over the document is employed and all word pairs in the window are connected. a limitation of graphbtm is that only word relationships are considered while ignoring document relationships. since a topic is possessed by a subset of documents in the corpus, we believe that the topical neighborhood of a document, i.e., documents with similar topics, would help determine the topics of a document. to this end, we propose graph topic model (gtm), a neural topic model that a corpus is represented as a document relationship graph where documents and words in the corpus are nodes and they are connected based on document-word co-occurrences. in gtm, the topical representation of a document node is aggregated from its multi-hop neighborhood, including both document and word nodes, using graph convolutional network (gcn) (kipf and welling, 2016). as gcn is able to capture high-order neighborhood relationships, gtm is essentially capable of modeling both word-word and doc-doc relationships. in specific, the relationships between relevant documents are established by their shared words, which is desirable for topic modeling as documents belonging to one topic typically have similar word distributions. the main contributions of the paper are: ## experiments the topic coherence results on the three datasets are shown in to explore how topic coherence results vary w.r.t. different topic numbers, we present in figure 2 the topic coherence scores under different topic numbers settings. it can be observed in figure 2 that gtm enjoys the best overall performance, achieving the highest scores in most settings. lda has a slightly higher npmi score on 20newsgroups dataset with 75 and 100 topics, nevertheless, gtm outperforms all baseline models with a relatively large margin on other settings of 20newsgroups. nvdm is apparently the worst-performing model, while performances of models other than gtm and nvdm are not so consistent. notably, w-lda, graphbtm, and lda obtain the second-best overall c p, c a, and npmi scores respectively. another observation from figure 2 is that gtm performs better on smaller topics, probably due to the fact that topics become more discriminative against each other when the topic number is small. to gain an intuitive impression on the discovered topics, we present in ## conclusion we have introduced graph topic model, a neural topic model that incorporates corpus-level neighboring context using graph convolutions to enrich document representations and facilitate the topic inference. both quantitative and qualitative results are presented in the experiments to demonstrate the effectiveness of the proposed approach. in the future, we would like to extend gtm to corpora with explicit doc-doc interactions, e.g., scientific documents with citations or social media posts with user relationships. replacing gcn in gtm with more advanced graph neural networks is another promising research direction.