Community Topic: Topic Model Inference by Consecutive Word Community Discovery (2022)

Abstract: We present our novel, hyperparameter-free topic modelling algorithm, Community Topic. Our algorithm is based on mining communities from term co-occurrence networks. We empirically evaluate and compare Community Topic with Latent Dirichlet Allocation and the recently developed top2vec algorithm. We find that Community Topic runs faster than the competitors and produces topics that achieve higher coherence scores. Community Topic can discover coherent topics at various scales. The network representation used by Community Topic results in a natural relationship between topics and a topic hierarchy. This allows sub- and super-topics to be found on demand. These features make Community Topic the ideal tool for downstream applications such as applied research and conversational agents.

## introduction topic modelling discovers the themes and concepts of large collections of unstructured text documents. these topics can fulfill multiple roles. they can act as features for document classification and indices for information retrieval. however, one of the most important functions of these topics is to assist in the exploration and understanding of large corpora. researchers in all fields and domains seek to better understand the main ideas and themes of document collections too large for a human to read and summarize. this requires topics that are interpretable and coherent to the human users. in more recent years, another new area has emerged where topics can provide a great deal of utility: conversational agents or "chat bots". a conversational agent is a computer program that is able to carry on a conversation with a human. the conversation is an end in itself; the purpose of speaking with a conversational agent is to converse, to be entertained, to express emotion and be supported. this goes well beyond asking siri to set a timer. one key component of having an actual conversation with a human is the awareness and use of the topic of conversation. work has been done on enriching the agent's response using the detected topic @xcite . however, more can be done with topics to improve a conversational agent given the right topic model. it can be used to detect and control topic drift in the conversation so that the agent's responses make sense in context. if the user is engaged with the current topic, then the agent can stay on topic or detect sub-topics to focus the conversation. the agent can detect super-topics to broaden the range of conversation. the agent should be able to move to related topics or, if the user becomes bored or displeased, jump to dissimilar topics. this type of control over the flow of the conversation is crucial to human communication and is needed for human-computer interaction. the features that make a topic model useful for a conversational agent are the same that make it useful as a tool of applied research. the topics must be coherent and interpretable to be useful to a researcher and for an agent's response to fit into a conversation. a measure of relatedness between topics allows for a natural flow to exploration and conversation. a natural hierarchical structure allows both a researcher and a conversational agent to drill down into more specific sub-topics or find broader super-topics on the fly. the most widely used topic model, latent dirichlet allocation (lda), lacks many of these features and has other drawbacks. the number of topics must be specified, requiring multiple runs with different numbers of topics to find the best topics. it performs poorly on short documents. different runs on the same corpus can produce different topics, especially if the order of the documents is different @xcite . common terms can appear in many different topics, reducing the uniqueness of topics @xcite . neural networks have recently pushed forward the state-of-the-art in topic modelling. while neural topic models have produced topics of greater coherence, they retain many of the weaknesses of lda, such as the need to specify the number of topics, while having a tendency to find models with many redundant topics @xcite and demanding greater computational resources and specialized hardware, i.e. gpus. these drawbacks have inspired us to search for an alternative approach to topic modelling, one that can operate quickly on commodity hardware and that provides not only a set of topics but their relationships and a hierarchical structure. given the growing importance of relational data and graphs in representing complex systems @xcite , it seems natural to take a network-based approach to topic modelling. our topic modelling algorithm, community topic (ct), mines communities from networks constructed from term co-occurrences. these topics are collections of vocabulary terms and are thus interpretable by humans. the network representation provides a natural topic structure and hierarchy. the topics themselves form a network with connections of varying strength between the topics and on which super-topics can be mined. each topic is a sub-graph that can be mined to find sub-topics. our algorithm can run quickly on simple hardware which makes it ideal for researchers from all fields for exploring a document collection. in this paper, we review related work on topic modelling. we describe our algorithm, how it constructs term co-occurrence networks, and how it mines topics from these networks. we empirically evaluate our algorithm and compare it to lda as a standard benchmark as well as a recently developed clustering approach based on word embeddings. our results show that our approach is able to find more coherent topics in a shorter period of time with more stable results while also providing a natural topic structure and hierarchy. ## related work topic modelling emerged from the field of information retrieval and methods for document indexing, query matching, and classification. the performance of topic models on these tasks has been surpassed by deep neural models but topic models have become extremely popular tools of applied research to better understand large document collections @xcite in fields as varied as political science @xcite and bioinfor-matics @xcite . one early approach was latent semantic analysis (lsa) @xcite . lsa decomposes the term-by-document matrix to find vectors representing the latent semantic structure of the corpus. these vectors relate terms and documents and can be viewed as topics, although they are uninterpretable. another method based on matrix decomposition is non-negative matrix factorization (nmf) @xcite . unsatisfied with the lack of a solid statistical foundation to lsa, researchers developed probabilistic latent semantic analysis (plsa) @xcite which has a generative probabilistic model of the data with the topics as the latent variables. one major drawback of plsa is that the topic mixture is estimated separately for each document. to remedy this, researchers developed latent dirichlet allocation (lda) @xcite . lda is also a hierarchical probabilistic model, but it is a fully generative model as it places a dirichlet distribution prior on the latent topic mixture of a document. the probability of a topic z given a document d, p(z|d), is a multinomial distribution over the possible topics parameterized by θ where θ is itself a random variable sampled from the prior dirichlet distribution parameterized by α. the generative process of a document is thus: the number of topics must be specified. lda maximizes the probability of the observed corpus assuming that it was generated by the hidden latent variables. this computation is intractable @xcite but can be approximated by variational inference @xcite or markov chain monte carlo @xcite . the topics are probability distributions over terms which are interpretable to human users. the trained model can discover the topic mix of unseen documents. while lda has been extremely successful and is widely used, there have been many attempts to improve upon it. researchers have tried promoting named entities to become the most frequent terms in the document @xcite . in @xcite , the authors use a two-step lda process to identify and re-weight words that are topic-indiscriminate. to improve the performance of lda on tweets, the authors of @xcite pool tweets into longer documents based on various schemes such as common author and same hashtag. the metalda model @xcite incorporates document and word meta information such as document labels, wordnet synonyms @xcite , and word embeddings @xcite . the author-topic model @xcite extends lda by conditioning the topic mixture on document author. the correlated topic model (ctm) @xcite ) models the correlations between topics. the dynamic topic model @xcite allows for the modelling of topic evolution over time. the hierarchical lda model (hlda) @xcite allows for a hierarchy of topics using a tree structure. a flexible generalization of lda is the pachinko allocation model (pam) @xcite . like hlda, pam allows for a hierachy of topics but this hierarchy is represented by a directed acyclic graph rather than a tree of fixed depth, allowing for a variety of relationships between topics and terms in the hierarchy. in recent years, new types of topic models have emerged based on neural networks and deep learning. some of these methods remain close to the lda framework while others are completely different approaches. the embedded topic model (etm) @xcite combines word embeddings trained using the continuous skip-gram algorithm @xcite with the lda probabilistic generative model. another approach is to use deep neural networks to learn the probability distributions of a generative probabilistic model. this can be done using a variational autoencoder (vae) @xcite . there have been many vae-based topic models developed, including the neural variational document model (nvdm) @xcite , the stick-breaking variational autoencoder (sb-vae) @xcite , prodlda @xcite , and dirichlet-vae @xcite . these models discover topics that are qualitatively different than those found by traditional lda, although there is debate as to whether they are truly superior @xcite . other approaches use the word embeddings learned by a neural network but do not use the probabilistic generative model framework. the top2vec algorithm @xcite clusters document vectors learned by the doc2vec algorithm @xcite . to find the topics for collections of related documents, first the dimensionality of the document embeddings is reduced to two dimensions using the umap algorithm @xcite . then dense clusters are found using hdbscan @xcite . the topic for a cluster of documents is the centroid of all those document vectors in the original embedding space and the most relevant terms are those whose embeddings are closest to the topic embedding. the approach closest to ours is vec2gc (rao and chakraborty, 2021). like top2vec, this algorithm uses doc2vec to learn document embeddings. vec2gc creates a network of the documents where edges exist between documents that have a cosine similarity over a certain threshold. community mining is then applied to the network to find communities of related documents. our approach differs in that it finds interpretable communities of terms, i.e. topics, rather than groups of similar documents. our approach does not rely on learning embeddings with a neural network and computing pairwise similarities but uses the co-occurrence information present in the documents themselves. ## community topic we call our community detection-based topic modelling algorithm community topic. the algorithm takes in a corpus of documents that have been preprocessed as described in section 3.3. first, a network is constructed from the document corpus. the user can select whether to use a sentence-based co-occurrence window or a sliding window of a fixed size. the user can also select whether to assign edge weights based on raw co-occurrence counts or npmi and whether to threshold the edge weights. as a practical matter, the npmi edge weights should at minimum be thresholded at 0 as negative edge weights cannot be handled by most community detection algorithms. very few of the npmi edge weights are negative so the impact of this thresholding on network structure is small. after the network is constructed, ct applies a community detection algorithm to find the communities in the network. communities of size 1 or 2 are filtered out as outlier terms that belong to no proper topic. finally, each topic (i.e. community) is sorted so that the most important and relevant terms for the topic come first and the topics are returned. algorithm 1 community topic require: preprocessed corpus d, parameters window, weight, threshold g ← buildnetwork(d, window, weight, threshold) communities ← communitydetection(g) t opics ← {} for @xmath0.length() > 2 then sort(community) topics.add(community) end if end for return t opics ## empirical evaluation we conduct empirical evaluations both to determine the best hyperparameters for ct as well as comparing community topic to two other topic modelling approaches, lda and top2vec. the code and data for the experiments in this section are available at a public github repository: https://github.com/eric-austin/topic_modelling . ## conclusion we have presented our novel topic modelling algorithm, community topic. we have conducted a thorough empirical evaluation of the algorithm to determine that it works best and needs no hyperparameter tuning with the leiden community detection algorithm. ct discovers topics with higher coherence scores than lda and top2vec. it is hyperparameter free and automatically discovers the number of topics with the user able to set the scale of the topics using the leiden resolution parameter. the discovered topics have a natural network hierarchy and relationships, allowing for the discovery of sub-and super-topics as desired. it is time and resource efficient, requiring no special hardware and discovering topics in less time than lda and top2vec and much less time than the hours of gpu training required by vae-based models @xcite . ct produces topics with no redundancy, a known issue with lda and neural topic models. these features make it an ideal tool for downstream applications such as conversational agents and corpus exploration by researchers. in the future, we will empirically evaluate ct against other models that provide a topic hierarchy. as many community detection algorithms failed to find quality topics on the co-occurrence networks, we will investigate ways to improve both the algorithms and the network representation. while automated coherence metrics give some idea of the topic quality, we plan to integrate ct into a conversational agent to truly test the coherence of the topics and the quality of the topic structure.