Topic Modeling Using Community Detection on a Word Association Graph (2023)

Abstract: Topic modeling of a text corpus is one of the most well-studied areas of information retrieval and knowledge discovery. Despite several decades of research in the area that begets an array of modeling tools, some common problems still obstruct automated topic modeling from matching users’ expectations. In particular, existing topic modeling solutions suffer when the distribution of words among the underlying topics is uneven or the topics are overlapped. Furthermore, many solutions ask the user to provide a topic count estimate as input, which limits their usefulness in modeling a corpus where such information is unavailable. We propose a new topic modeling approach that overcomes these shortcomings by formulating the topic modeling problem as a community detection problem in a word association graph/network that we generate from the text corpus. Experimental evaluation using multiple data sets of three different types of text corpora shows that our approach is superior to prominent topic modeling alternatives in most cases. This paper describes our approach and discusses the experimental findings.

## introduction the goal of topic modeling is to find the underlying semantic structure in a corpus that succinctly describes the documents and the text forming the corpus without compromising the corpus's statistical characteristics. it is one of the oldest and most researched problems in the field of information retrieval and has numerous direct and downstream applications such as document grouping, classification, retrieval, and summarization. for a long time, the most prominent solutions to topic modeling are latent dirichlet allocation (lda) @xcite and its variants. lda works under the principle of 'interchangeability' of documents and describes a document as a random mixture of a fixed number of latent 'topics' drawn from a dirichlet distribution where each topic is a distribution of words. lda is a bag of words model as it disregards the position of words in a document. lda's assumption of a latent dirichlet distribution for the topics and interchangeability of documents under the bag of words model is both for the mathematical tractability of the problem, as its authors admit @xcite , as opposed to its conformance with any empirical rule describing real texts (e.g., the zipf's law @xcite ). however, lda provides the first principled approach to group both the documents and the words of a corpus and remains widely applicable. consequently, most subsequent works on topic modeling focused on improving the lda model which led to many lda variants. for example, biterm @xcite adapts lda for short texts, hdp @xcite ) eliminates lda's requirement of an input topic count, and seededlda @xcite incorporates human input in lda training. most recent neural network solutions for topic modeling also focus on tackling specific shortcomings of lda as opposed to proposing a better alternative mathematical foundation. for example, prodlda @xcite addresses the concern of difficulty of gibb's sampling and variational inference for lda training by transferring the model parameters to the neural space, etm @xcite ) addresses lda's limitations in dealing with sparse and large vocabularies by using word embeddings, and ctm @xcite generalizes lda for cross-lingual topic modeling. all these solutions improve lda in different respects but some fundamental limitations of lda persist and hurt its effectiveness in many scenarios. a frequently cited problem with lda and its variants is the large discrepancy between their output and human judgment @xcite . various attempts are being made to overcome this problem by making lda-based topic modeling semi-supervised, e.g., in itm @xcite , guidedlda @xcite , and sshlda @xcite . all these solutions apply some human-provided constraints on the lda model training and only attain partial success as the document collections may not fit a latent dirichlet distribution in the first place @xcite . then given lda tries to fit a probabilistic generative model against a corpus using maximum likelihood estimation or some other statistical measure, it tends to overlook small topics @xcite and struggles when topics are overlapped @xcite . in this paper, we present an alternative to ldabased topic modeling to address the above problems using a document-structure-sensitive topic modeling through community detection @xcite in a word co-occurrence graph. we call our solution comtm foot_0 . in comtm, we use the structure of the documents in the text corpus to generate the co-occurrence graph with words that capture the core information flow of the documents, as opposed to all words. then we apply a novel overlapping community detection algorithm to extract the topics. comtm is applicable when the type/source of the documents in the collection is uniform and known. this assumption is practical as topic modeling is frequently applied to a corpus of a specific type of document such as only news articles, scientific publications, or wikipedia articles. meanwhile, the assumption is necessary to apply a single strategy to capture the information flow of the documents. when the documents are of manifold type, it is unrealistic to assume the user knows their information flow structures. comtm is not the first attempt to apply community detection to the topic modeling problem. community detection is a widely studied branch of network science that discovers meaningful clusters/communities in a graph by analyzing its wiring diagram @xcite . it shows significant success in describing graphs originating from biological, physics, and human networks and begets several popular algorithms. the particular appeal of community detection is that it does not require any cluster/community count as input which was a major obstacle for the application of traditional graph partitioning algorithms @xcite in realworld networks. another important characteristic of community detection algorithms is that they are non-parametric and can detect communities when their composition in the network is an uneven mixture of small and large communities. these attractive features led researchers to apply community detection for topic modeling of text corpora. for example, community detection has been applied to guide lda topic modeling using network-structured metadata such as citation information @xcite . some recent works completely replace lda with community-detection-based topic modeling using a different statistical criterion for community fitness calculation @xcite , called minimum description length (mdl) @xcite , that originates from information theory. however, to the best of our knowledge, no existing solutions apply overlapping community detection for which existing algorithms have exponential or inordinately high-degree polynomial running times, otherwise producing poor results. secondly, none of them incorporates any notion of differential importance of words of the documents when constructing the word co-occurrence graphs. comtm's ingenuity lies in these two aspects. we compared comtm with several lda variants and a prominent community-detection-based topic model, called hsbm @xcite , on several data sets. the data sets are constructed or collected from online news articles from several outlets, scientific papers, and wikipedia articles. we evaluated the topic models' output using both human annotators and cluster coherence measurement. the result shows that comtm consistently outperforms other solutions for most data sets and for overlapping data sets in particular. this paper discusses our experimental findings along with the design methodology and algorithms for comtm. to summarize, the contributions of this paper are as follows: ## design methodology in designing comtm, we apply a notion/framework of document 'interchangeability' quite different from lda. in our framework, the places a word occurs in a document are significant. words in a document attain importance by virtue of their inclusion in larger semantic structures, such as sentences or paragraphs, that carry the core information of the document. in other words, we interpret words occurring in more important sentences/paragraphs as more important than other words. under this interpretation, a pair of documents are more or less interchangeable depending on the similarity of their text contents in parts that carry the central information the documents try to convey. consequently, the collection comtm can process must contain documents that are structurally similar and whose structure reflects the relative importance of different semantic blocks within the documents. evidently, comtm is not a generic topic model yet as its applicability relies on a preprocessing step that can explain the structure of an arbitrary document in terms of the relative importance of different parts. the current scope of comtm covers three document classes: 'hard' news articles, scientific papers, and wikipedia articles. hard news articles are those that report on recent incidents of local and global importance (lehman-wilzig and seletzky, 2010) (thus, opinion pieces, interviews, and long-read articles are not hard news). they constitute the majority of daily news publications worldwide @xcite @xcite . historically hard news articles follow an inverted pyramid model to capture the short attention span of typical news readers. in this model, the leading paragraph summarizes the key points of the event that subsequent paragraphs elaborate on in decreasing order of importance. numerous studies from media sociology validate adherence to this structure in english news articles @xcite . next we use scientific articles as they include an abstract section that abridges the contribution of the paper, which allow us to treat the abstract as a container of some of the most important words in this document category. finally, we target wikipedia articles as generic multi-section descriptive prose category and apply and concatenate extractive summaries of those sections to form the most semantically significant document part. we trust on extractive summaries for wikipedia articles because the sota tools for summarization are frequently trained on wikipedia data and extractive summaries are generated following a theory of information contribution of sentences to the meaning of their containing document. we then construct a word co-occurrence (aka, association) graph using the words of the central information-bearing part of the documents. in this graph, the nodes are words and there is an edge between two nodes if the corresponding words appear in the same document -not necessarily in its central part -anywhere in the corpus. the graph is weighted, where the weight of an edge reflects in how many documents the corresponding pair of words co-occurred. then we apply a non-overlapping community detection on the word-occurrence graph. the goal here is to partition the graph into clusters. however, unlike the other community-detection-based topic models that consider the discovered communities to be topics and the most frequent words in the communities as the top topic words @xcite , we apply eigenvector centrality measure to filter the most important words from identified communities. eigenvector centrality encapsulates other notions of graph centrality such as between-ness, degree, and closeness centrality @xcite and considers edge weights, which community detection ignores. there is a philosophical reason for choosing this alternative significance measure also that the eigenvector centrality measure reflects better: there is no reason to assume that the item which recurs most frequently is the most important ... the place occupied by the different elements is more important than the number of times the recur. oliver burgelin @xcite in essence, we apply a standard community detection algorithm on the word occurrence graph to get the topic count and identify the central words of the individual topics. subsequently, we construct a larger word co-occurrence graph by considering all words in the corpus and applying our own algorithm to associate other words with the central topic words based on a graph proximity calculation. at that time a single word can be associated with multiple topics. this two-step process can be described as a new overlapping community detection algorithm for a weighted word co-occurrence graph that returns topics as word distributions. ## algorithm & implementation although there is evidence that community detection algorithms can handle word co-occurrence graphs formed from unfiltered text corpora, we removed stop word and lemmatized in comtm on the ground of pragmatism. in addition, we only considered nouns and verbs in the initial graph that comtm uses for the topic count and central topic word identification. in that regard, comtm uses the nltk @xcite package for parts of speech tagging. in addition, we apply word-net @xcite super-subordinate relation among the filtered words and replace them with their immediate hypernyms. this has been done to capture the clustering tendency among the words in the co-occurrence graph at a higher conceptual level and also to make the graph more compact when the corpus is large. for hard news articles and scientific papers, the word set each document contributes to the cooccurrence graph comes from the leading paragraph and the abstract section respectively. for wikipedia articles, comtm uses the bert based extractive summarizer @xcite for each section then concatenates the summary of the sections. we found that the summaries incorporate most keywords of the documents. still, we added the output of the keybert keywords extractor @xcite in the word set of the combined summary in comtm's initial word co-occurrence graph process. finally, comtm drops words from the sets that occurred in only a single document before the graph construction as they cannot influence the community structure of the corpus but increase the memory and processing footprint of the community detection algorithm. ## experiments since the qualitative value of found topics under human judgment is comtm's main target, statistical measures such as perplexity score or maximal likelihood commonly used for evaluating topic models @xcite are of little use. earlier research shows that these measures do not typically correlate with human judgment @xcite . therefore, you employed five human annotators to judge the topic outputs of comtm and reference baseline implementations. we estimated the iaa(inter-annotator agreement) of the annotators using fleiss' kappa @xcite to assess the quality of the annotations. the score-0.4275 indicates that the human judgments were highly similar among the five annotators. we compared comtm with lda @xcite , ctm @xcite , etm @xcite , hdp @xcite , prodlda (srivastava and sutton, 2017b), hsbm @xcite and seeded-lda @xcite . for reference implementations of these existing topic models, we use gensim @xcite and octis @xcite libraries. we used the graph-tool library (peixoto, 2014) for visualizing the comparison results. however, we applied two techniques to avoid making our evaluation completely subjective. first, we compared cluster coherence scores @xcite of different topic model outputs in each data set. some empirical studies show cluster coherence scores for frequent words correspond well with human judgment. second, we use curated datasets with known categories of documents (e.g., sports, business, and politics can be different categories of a hard news dataset) for various experiments to assess the topics' relevance to those categories. ## conclusion in this research, we proposed comtm, a topic model based on community detection. comtm is document structure sensitive and does not require a preset topic count as input. our experiments on multiple datasets of hard news articles, scientific abstracts and wiki data show that comtm is generally superior to the dominant topic modeling alternatives in this particular domain. given that comtm utilizes the structure of documents to identify core words in each document as a pre-processing step in its modeling, alternative mechanisms to core words/terms identification augmented with comtm has prospect for improvement in topic modeling in other domains as well. we encourage other researchers to investigate this prospect.