Unsupervised Hierarchical Topic Modeling via Anchor Word Clustering and Path Guidance (2024)

Abstract: Hierarchical topic models nowadays tend to capture the relationship between words and topics, often ignoring the role of anchor words that guide text generation. For the first time, we detect and add anchor words to the text generation process in an unsupervised way. Firstly, we adopt a clustering algorithm to adaptively detect anchor words that are highly consistent with every topic, which forms the path of topic \rightarrow anchor word. Secondly, we add the causal path of anchor word \rightarrow word to the popular Variational Auto-Encoder (VAE) framework via implicitly using word co-occurrence graphs. We develop the causal path of topic+anchor word \rightarrow higher-layer topic that aids the expression of topic concepts with anchor words to capture a more semantically tight hierarchical topic structure. Finally, we enhance the model’s representation of the anchor words through a novel contrastive learning. After jointly training the aforementioned constraint objectives, we can produce more coherent and diverse topics with a better hierarchical structure. Extensive experiments on three datasets show that our model outperforms state-of-the-art methods.

## introduction topic models, which can automatically discover coherent and meaningful topics from text corpora, have been widely used for text analysis @xcite @xcite . in such methods, each topic is interpreted as relevant words to represent a semantic concept. different from traditional flat topic models, hierarchical topic models (htms) aim to leverage the hierarchical nature of topics to build a rational topic structure @xcite . htms have been successfully applied to tasks such as hierarchical classification of web pages @xcite and the discovery of hierarchical relationships in academic repositories @xcite . * the corresponding author. existing htms can be divided into two categories. the first category is conventional models like hlda @xcite and its variants @xcite . they infer parameters through gibbs sampling or variational inference, which require high computational costs or complex derivation @xcite . the second category is neural hierarchical topic models, including hntm @xcite , hyperminer @xcite , and so forth @xcite @xcite @xcite . these methods generally follow the vae framework and employ back-propagation for faster parameter inferences @xcite . however, the generation process of most previous methods overlooks the role of anchor words and directly generates words through topics, resulting in insufficient mining of fine-grained topic-word information. in this study, we adopt the framework of the second category and attempt to alleviate the above issue by exploiting anchor words. the anchor word guided method @xcite is based on the separability assumption @xcite , which assumes that each topic contains at least one highly relevant anchor word that uniquely identifies the topic. for example, while "resurrection", "pray", "sin", and "christ" are associated with a topic about christianity, only "christ" is unambiguous, so it could serve as this topic's anchor word. based on this assumption, there are two types of anchor word guided topic models. the first type detects anchor words by an interactive process, and uses these words to identify topics in the context of non-negative matrix factorization @xcite @xcite . these works require intensive matrix calculations, making them unstable and noisy @xcite . the second type obtains anchor words through label information @xcite @xcite . these works can be seen as a semi-supervised anchor word detection fashion, which is unachievable without any external information. although aforementioned works demonstrate the important role of anchor words in guiding topicword relationships, they have their own shortcomings and the application of anchor words in unsupervised hierarchical topic mining is scarce. so there are two key issues here: first, how to detect anchor words unsupervisedly on a neural hierarchical topic model; second, how anchor words can help model fine-grained topic-word relationships and hierarchical topic structures. in our proposed anchor word clustering and path guided framework for unsupervised hierarchical topic modeling (ap-htm), we detect anchor words and introduce four causal paths to constrain the text generation process. fig. 1(a) and @xcite show the text generation processes of our ap-htm and other htms @xcite @xcite , respectively. first, we unsupervisedly detect the anchor words of each topic by a clustering algorithm @xcite , which forms the path of @xmath0, as shown in fig. 1(c ). second, two causal paths related to the anchor words are added to the text generation process. in order to obtain a semantically tight final embedding space, the causal path of anchor @xmath1. 1(d )) is introduced in each layer. we use the word co-occurrence graph to implicitly capture the relationships between anchor words and other words, e.g., "christ"→"scriptures", "bible". then, the topic generation structure of topic+anchor @xmath2. 1(e) ) is adopted between layers to capture the topic hierarchical relationships, utilizing anchor words as auxiliary information to enrich the conceptual relevance between topics. further, for decoupling of anchor words and other words in causal diagram, we employ a novel contrastive learning (fig. 1(f )), masking anchor words during inference that makes the encoder pay more attention to the vital information of anchor words. in summary, our contributions are as follows: ## background we define the basic hierarchical topic modeling process in the following parts. encoder: given a collection of documents, we process each document d into a bag-of-words (bow) vector x @xmath3, where v is the vocabulary size. the gaussian mixture encoder network can be described as follows: decoder: the document decoder can be expressed as follows: ## method after introducing the basic hierarchical topic modeling process in section 3, we briefly describe four paths that extend the basic vae framework in section 4.1, which are introduced to model finegrained topic-word relationships and the hierarchical structure. in section 4.2 and section 4.3, we present the specifics of how our model works in the form of two modules (fig. 2 ). section 4.4 shows the training details of our ap-htm. ## conclusion in this paper, we propose a neural hierarchical topic model ap-htm based on anchor words. unlike the popular prior models, our approach adds the concept of anchor words to the text generation process. we obtain the anchor space by clustering and propose three causal paths guided by anchor words to extend the vae framework. in addition, we introduce a novel anchor-based contrastive learning to decouple the roles of anchor words in the paths, thus endowing the model with stronger anchor word and topic representation. extensive experiments show that our framework outperforms stateof-the-art baselines in extracting coherent, unique, and rationally structured topics.