GIN opic: Topic Modeling with Graph Isomorphism Network (2024)

Abstract: Topic modeling is a widely used approach for analyzing and exploring large document collections. Recent research efforts have incorporated pre-trained contextualized language models, such as BERT embeddings, into topic modeling. However, they often neglect the intrinsic informational value conveyed by mutual dependencies between words. In this study, we introduce GINopic, a topic modeling framework based on graph isomorphism networks to capture the correlation between words. By conducting intrinsic (quantitative as well as qualitative) and extrinsic evaluations on diverse benchmark datasets, we demonstrate the effectiveness of GINopic compared to existing topic models and highlight its potential for advancing topic modeling.

## introduction the rise in digital text data makes organizing them manually by theme increasingly difficult. topic modeling plays a significant role here @xcite @xcite , as it can uncover the underlying topics in documents in an unsupervised manner. in topic modeling, we assume that each document is a mixture of topics and these latent topics are also defined as distribution over the words. motivation: recent approaches to neural topic modeling @xcite focus on the representation of the document as a sequence of words, which captures the contextual information. however, words in a document may be correlated to each other in a much more complex manner. so, why not explicitly consider these word dependency patterns while learning the topics? several studies in the field of topic modeling delve into the representation of documents using graphs. in this context, nodes signify words, and edges depict relationships between words, such as syntax or semantic relations. for instance, in the case of short texts, the graph biterm topic model (graphbtm) @xcite , an extension of the biterm topic model (btm) @xcite , represents word co-occurrence as a graph, with nodes representing words and weighted edges reflecting the counts of corresponding biterms. despite graphbtm's emphasis on capturing word dependencies, it has been reported to exhibit poor performance @xcite . additionally, its computational cost escalates with an expanding vocabulary, as it constructs a single graph using the entire vocabulary. in contrast, the graph neural topic model (gntm) @xcite employs a directed graph with word dependencies as edges between word nodes to incorporate semantic information from words in documents. however, gntm considers word dependency solely by linking words within a small sliding window for a given document. this limitation makes it impossible to account for word dependencies that fall outside of that specific window. furthermore, the computational complexity of generating document graphs increases with the length of the window. approach: to model the mutual dependency between words while addressing the existing issues of incorporation of document graphs into topic modeling, we developed a neural topic model that takes the word similarity graphs for each document, where the word similarity graph is constructed using word embeddings to capture the complex correlations between the words. these document graphs along with their unordered frequency-based text representation are then used as input. we have also used the graph isomorphism network (gin) to obtain the representation for each document graph. we have used gin as it is provably the maximally powerful gnn under the neighborhood aggregation framework. it is as powerful as the weisfeiler-lehman graph isomorphism test @xcite . contributions: in summary, our work presents the following key contributions: ## related work topic modeling processes extensive document collections efficiently, preserving key statistical relationships for tasks like classification, novelty detection, summarization, and similarity judgments. traditional models like latent dirichlet allocation (lda) @xcite , probabilistic latent semantic index (plsi) @xcite , and correlated topic model (ctm) @xcite use sampling-based algorithms or variational inference, but their design is limited by the need for careful selection, which limits the flexibility and scalability of model design. recent advancements in neural variational inference, particularly auto encoding variational bayes (aevb) (kingma and welling, 2014), simplify posterior computation. neural variational document model (nvdm) @xcite is the pioneer vae-based topic model. however, following the traditional topic models of applying dirichlet-prior to the document-topic distribution becomes challenging due to the limitations of the reparametrization trick. autoencoding variational inference for topic models (avitm) @xcite resolves this by using laplacian approximation of the dirichlet parameter with gaussian parameters. combinedtm @xcite extends avitm by incorporating sentence bert embeddings alongside bag-of-words (bow) representations. zeroshottm @xcite further extends this approach, relying solely on sbert embeddings, ignoring word co-occurrence relations in input documents. numerous contemporary methodologies incorporate graph neural networks (gnns) for topic modeling. in terms of the graph construction task, the graph biterm topic model (graphbtm) @xcite and the graph neural topic model (gntm) @xcite employ a moving window-based approach with a specified window length to model word co-occurrence relationships, necessitating careful window length selection. the graph topic model (zhou et al., 2020) constructs document graphs based on tf-idf scores, capturing relationships with graph convolutions. topic modeling with knowledge graph embedding @xcite incorporates external knowledge graphs. the graph attention topic network @xcite addresses overfitting in probabilistic latent semantic indexing with amortized inference and word embeddings. the graph relational topic model @xcite explores document relationships using higher-order graph attention networks. ## proposed methodology recognizing the challenges in topic modeling, we acknowledge the necessity of capturing semantic similarity among words in a document. additionally, we note the importance of addressing the graph construction issue and obtaining unique representations for dissimilar document graphs. in response to these challenges, we have introduced the graph isomorphism network-based neural topic model, abbreviated as ginopic. the following subsections provide a detailed explanation of the graph construction methodology, model framework, and objective function. ## experimental settings we have conducted the experiments using octis foot_0 @xcite , a comprehensive framework for comparing and optimizing topic models, available under the mit license. ## results and discussions we categorize our findings into the following sections: (1) quantitative evaluation (section 5.1), (2) extrinsic evaluation (section 5.2), (3) qualitative evaluation (section 5.4), (4) latent space visualization (section 5.3), and (5) sensitivity analysis (section 5.5). ## conclusion we have introduced ginopic, a neural topic model based on a graph isomorphism network, and evaluated its performance on five widely used datasets for assessing topic models. across the majority of our experiments, ginopic consistently exhibits superior topic coherence and diversity compared to other competitive topic models from the literature. manual evaluation of selected topics further confirms that ginopic generates more coherent topics than alternative models. in extrinsic evaluations, ginopic generally outperforms existing models across all the datasets, except for the 20ng dataset. we utilized visualizations of the latent space generated by ginopic to assess its clustering disentanglement capability. sensitivity analysis demonstrates the impact of graph construction threshold values on the performance and training time of ginopic. additionally, we highlight the effectiveness of gin over other graph neural networks in our topic model.