H y HTM : Hyperbolic Geometry-based Hierarchical Topic Model (2023)

Abstract: Hierarchical Topic Models (HTMs) are useful for discovering topic hierarchies in a collection of documents. However, traditional HTMs often produce hierarchies where lower-level topics are unrelated and not specific enough to their higher-level topics. Additionally, these methods can be computationally expensive. We present HyHTM - a Hyperbolic geometry-based Hierarchical Topic Model - that addresses these limitations by incorporating hierarchical information from hyperbolic geometry to explicitly model hierarchies in topic models. Experimental results with four baselines show that HyHTM can better attend to parent-child relationships among topics. HyHTM produces coherent topic hierarchies that specialize in granularity from generic higher-level topics to specific lower-level topics. Further, our model is significantly faster and leaves a much smaller memory footprint than our best-performing baseline. We have made the source code for our algorithm publicly accessible.

## introduction the topic model family of techniques is designed to solve the problem of discovering humanunderstandable topics from unstructured corpora @xcite where a topic can be interpreted as a probability distribution over words @xcite . hierarchical topic models (htms), in addition, organize the discovered topics in a hierarchy, allowing them to be compared with each other. the topics at higher levels are generic and broad while the topics lower down in the hierarchy are more specific @xcite . while significant efforts have been made to develop htms @xcite @xcite , there are still certain areas of improvement. words such as space shuttle and satellite, which belong to moderately different super-concepts such as vehicles and space, respectively, are brought closer together due to their semantic similarity. this leads to a convergence of their surrounding words, such as helicopter and solar system, creating a false distance relationship and a crowding effect in euclidean spaces. in figure (b), we see a concept tree in hyperbolic spaces (poincaré ball), which inherently has more space (represented by grey circles) than euclidean spaces. the distances here grow exponentially towards the edge of the ball, and the concepts at deeper levels such as helicopter and solar systems move apart in these growing spaces and are far from each other. the dashed blue line shows how the distances in both spaces are calculated. first, the ordering of topics generated by these approaches provides little to no information about the granularity of concepts within the corpus. by granularity, we mean that topics near the root should be more generic, while topics near the leaves should be more specific. second, the lower-level topics must be related to the corresponding higher-level topics. finally, some of these approaches such as cluhtm @xcite ) are very computationally intensive. we argue that these htms have such shortcomings primarily because they do not explicitly account for the hierarchy of words between topics. most of the existing approaches use document representations that employ word embeddings from euclidean spaces. these spaces tend to suffer from the crowding problem which is the tendency to accommodate moderately distant words close to each other @xcite . there are several notable efforts that have shown that euclidean spaces are suboptimal for embedding concepts in hierarchies such as trees, words, or graph entities @xcite @xcite . in figure 1(a), we show the crowding of concepts in euclidean spaces. words such as space shuttle and satellite, which belong to moderately different concepts such as vehicles and space, respectively, are brought closer together due to their semantic similarity. this also leads to a convergence of their surrounding words, such as helicopter and solar system creating a false distance relationship. as a result of this crowding, topic models such as cluhtm that use euclidean word similarities in their formulation tend to mix words that belong to different topics. contrary to this, hyperbolic spaces are naturally equipped to embed hierarchies with arbitrarily low distortion @xcite @xcite . the way distances are computed in these spaces are similar to tree distances, i.e., children and their parents are close to each other, but leaf nodes in completely different branches of the tree are very far apart @xcite . in figure 1(b), we visualise this intuition on a poincaré ball representation of hyperbolic geometry (discussed in detail in section 3). as a result of this tree-like distance computation, hyperbolic spaces do not suffer from the crowding effect and words like helicopter and satellite are far apart in the embedding space. inspired by the above intuition and to tackle the shortcomings of traditional htms, we present hy-htm, a hyperbolic geometry based hierarchical topic model which uses hyperbolic geometry to create topic hierarchies that better capture hierarchical relationships in real-world concepts. to achieve this, we propose a novel method of incorporating semantic hierarchy among words from hyperbolic spaces and encoding it explicitly into topic models. this encourages the topic model to attend to parent-child relationships between topics. experimental results and qualitative examples show that incorporating hierarchical information guides the lower-level topics and produces coherent, specialised, and diverse topic hierarchies (section 6). further, we conduct ablation studies with different variants of our model to highlight the importance of using hyperbolic embeddings for representing documents and guiding topic hierarchies figure 2 : comparing our hyperbolic-based hyhtm model to the euclidean-based cluhtm, for selected 20news document labels (comp.graphics, comp.os.ms-windows.misc), we find that hyhtm is better at discriminating between similar document labels. cluhtm's root-level topics are not related to computer concepts, and it cannot separate these labels at lower levels. hyhtm groups them in the same root level and separates them into different lower-level topics, showing the advantage of using hyperbolic embeddings over euclidean ones to avoid the crowding problem. we show the top words with the highest probability for the topics. ## related work to the best of our knowledge, htms can be classified into three categories, (i) bayesian generative model like hlda @xcite , and its variants @xcite @xcite utilize bayesian methods like gibbs sampler for inferring latent topic hierarchy. these are not scalable due to the high computational requirements of posterior inference. (ii) neural topic models like tsntm @xcite and others @xcite use neural variational inference for faster parameter inference and some heuristics to learn topic hierarchies but lack the ability to learn appropriate semantic embeddings for topics. along with these methods, there are (iii) non-negative matrix factorization (nmf) based topic models, which decompose a term-document matrix (like bag-of-words) into low-rank factor matrices to find latent topics. the hierarchy is learned using some heuristics @xcite or regularisation methods @xcite based on topics in the previous level. however, the sparsity of the bow representation for all these categories leads to incoherent topics, especially for short texts. to overcome this, some approaches have resorted to incorporating external knowledge from knowledge bases (kbs) @xcite or leveraging word embeddings @xcite . pre-trained word embeddings are trained on a large corpus of text data and capture the relationships between words such as semantic similarities, and concept hierarchies. these are used to guide the topic hierarchy learning process by providing a semantic structure to the topics. @xcite utilizes euclidean embeddings for learning the topic hierarchy. however, @xcite ; @xcite ; @xcite ; @xcite have shown how the crowding problem in euclidean spaces makes such spaces suboptimal for representing word hierarchies. these works show how hyperbolic spaces can model more complex relationships better while preserving structural properties like concept hierarchy between words. recently, shi xu et al. made an attempt to learn topics in hyperbolic embedding spaces. contrary to the htms above, this approach adopts a bottom-up training where it learns topics at each layer individually starting from the bottom, and then during training leverages a topic-linking approach from @xcite , to link topics across levels. they also have a supervised variant that incorporates concept hierarchy from kbs. our approach uses latent word hierarchies from pretrained hyperbolic embeddings to learn the hierarchy of topics that are related, diverse, specialized, and coherent. ## preliminaries we will first review the basics of hyperbolic geometry and define the terms used in the remainder of this section. we will then describe the basic building blocks for our proposed solution, followed by a detailed description of the underlying algorithm. ## hierarchical topic models using hyperbolic geometry we now describe hyhtm -our proposed hyperbolic geometry-based hierarchical topic model. we first describe how we capture semantic similarity and hierarchical relationships between terms in hyperbolic space. we then describe the stepby-step algorithm for utilizing this information to generate a topic hierarchy. ## experimental results in this section we compare our model's performance on well-estabilished metrics to assess the coherence, specialisation, and diversity of topics. we present qualitative comparision for selected topics in figure 2 and in appendix 6.2. we discuss and perform ablations to show the role of hyperbolic spaces and effectiveness of our algorithm (see appendix 7). rq1: does hyhtm produce coherent topics? topic coherence is a measure that can be used to determine how much the words within a topic cooccur in the corpus. the more the terms co-occur, the easier it is to understand the topic. we employ the widely used coherence measure from @xcite and report the average across the top 5 and 10 words for every topic in ## ablation do hyperbolic embeddings represent documents better than euclidean ones? to investigate this we consider a variant of our model called ours (euc) which incorporates pretrained fasttext @xcite (trained on euclidean spaces) instead of poincare embeddings in m s (w, w ′ ), and we keep all the other steps unchanged. from we examine this by comparing the ours (euc) variant and the cluhtm baseline. both models use identical underlying document representations, yet they differ in how they guide their hierarchies, particularly in the equation 8 of our model. as demonstrated in in the appendix (see section b), we also examine the importance of our approach by replacing the underlying algorithm with hierarchical clustering methods. ## conclusion in this paper, we have proposed hyhtm, which uses hyperbolic spaces to distill word hierarchies of higher-level topics in order to refine lower-level topics. both quantitative and qualitative experiments have demonstrated the effectiveness of hyhtm in creating hierarchies in which lower-level topics are realted and more specific than higher-level topics. hyhtm is much more efficient compared to our best-performing baseline. a major limitation of hyhtm is that it is parametric and therefore requires empirical analysis to find the optimal number of topics at each level. we plan to investigate this shortcoming in the future.