Scale-Invariant Infinite Hierarchical Topic Model (2023)

Abstract: Hierarchical topic models have been employed to organize a large number of diverse topics from corpora into a latent tree structure. However, existing models yield fragmented topics with overlapping themes whose expected probability becomes exponentially smaller along the depth of the tree. To solve this intrinsic problem, we propose a scale-invariant infinite hierarchical topic model (ihLDA). The ihLDA adaptively adjusts the topic creation to make the expected topic probability decay considerably slower than that in existing models. Thus, it facilitates the estimation of deeper topic structures encompassing diverse topics in a corpus. Furthermore, the ihLDA extends a widely used tree-structured prior (Adams et al., 2010) in a hierarchical Bayesian way, which enables drawing an infinite topic tree from the base tree while efficiently sampling the topic assignments for the words. Experiments demonstrate that the ihLDA has better topic uniqueness and hierarchical diversity thanexisting approaches, including state-of-the-art neural models.

## introduction topic models @xcite @xcite have been used to summarize, annotate, and categorize documents. recent advances in large-scale topic models have enabled the estimation of thousands of topics to accommodate various concepts in a large corpus @xcite @xcite , requiring users to interpret numerous topics. hierarchical topic models have been proposed to improve the topic organization by learning the latent topic hierarchy @xcite @xcite @xcite @xcite . however, these hierarchical topic models will create a fragmented tree structure with the probabilities of a substantial number of topics becoming exponentially smaller. these topics typically have few assigned words and similar word distributions. recent hierarchical topic models with neural architectures @xcite have the same issue of topic fragmentation and use a fixed number of layers for all documents @xcite . the reason for the topic fragmentation is that the stick-breaking process @xcite used in existing models creates topics whose expected probability decays along the depth of the tree. existing models alleviate the issue by restricting the tree structure, for example by truncating the depth to three levels. @xcite also introduced a topic-diversity regularizer and a heuristic rule to update topics, whereas @xcite truncated topics based on their corpus coverage. to address this intrinsic issue of topic probabilities, we propose a scale-invariant hierarchical infinite topic model (ihlda) and make three main contributions. first, the ihlda adjusts the probability scale of the stick-breaking process at each level by considering the size of the parent topic to avoid fragmented topic structures. the expected topic probability of the ihlda decays considerably slower than that of the existing models, thereby reflecting the diversity of topics in a corpus by using a flexible depth and width. the existing probabilistic and neural models that leverage the stick-breaking process can also benefit from our model. second, the ihlda extends the tree-structured stick-breaking process (tssb; @xcite , figure 1 : overview of the tree-structured stick-breaking process (tssb) in @xcite . the blue intervals represent topics with probabilities proportional to their widths. a path of a topic is denoted in square brackets, and the solid lines show connections between topics. the right-hand tree recasts the topic structure. ψ's are the horizontal probabilities of breaking the stick, and each π denotes the probability of using a topic. ## background: tree-structured stick-breaking process a tree-structured stick-breaking process (tssb) @xcite is a prior for constructing a topic tree of theoretically unbounded depth and width, comprising two types of stick-breaking processes @xcite . figure 1 illustrates a draw from the tssb, where each blue interval represents a topic, whereas the square brackets denote the path to reach it. 1 hierarchical topic models assign a latent topic to each word in a document. as an equivalent representation of the dirichlet process (see appendix a for details), a stickbreaking process repeatedly breaks a stick of length 1, where each broken stick corresponds to a topic with the length equal to its probability. appendix b provides a formal definition and illustration of the stick-breaking process. here, we introduce notation to formalize the tssb. topic ϵ at the level |ϵ| in a tree has its ancestors and children. let κ ≺ ϵ indicate that κ is an ancestor of ϵ: in figure 1 , topic [1 1 1] has two ancestors, {κ : κ ≺ [1 1 1]} = {[1], [1 1]}. specifically, we use a prime symbol ′ to denote the parent topic, i.e., [1 1 1] ′ = [1 1]. the child topics of ϵ are {ϵk : @xmath0, 2, 3, . . .}. for example, topic [1 2] in figure 1 given this setup, the probability assigned to a topic ϵ under tssb can be expressed as a product of stick-breaking processes: the first term in equation ( 1 ) is the probability of stopping at the topic ϵ vertically. the next product terms refer to passing ancestors of ϵ while horizontally stopping at ϵ and its ancestors. these vertical and horizontal probabilities of stopping follow beta distributions: we also introduce a scaling factor λ used in @xcite and set α 0 at each level, @xcite . this parametrization makes a word more likely to stop as |ϵ| becomes larger, i.e., deeper in the tree. hereafter, we do not use subscript ϵ and denote α ϵ as α for simplicity. ## scale-invariant tssb although the tssb constitutes a crucial building block of recent hierarchical topic models @xcite , the expected probability of each topic in the tssb decays exponentially along the depth of the topic hierarchy. figure 2 (a) shows two topic trees drawn from the original tssb: topics in the third and the fourth levels have extremely small probabilities compared to the topics in the higher levels, resulting in a topic fragmentation in the tree. this property of the tssb is attributed to the probability of a horizontal stop, ψ ϵ , having the same expectation regardless of the level. as shown in appendix d, the expected probability of a horizontal break at the level ℓ is e[ϕ|ℓ] ≈ 1/(2γ +1) ℓ , where the level appears in the exponent of the denominator. the dotted line in figure 3 depicts this exponential decay with ℓ. to avoid this exponential decay, we rescale γ 0 in equation (2): the key idea in equation ( 3 ) is to use the horizontal breaking proportion of a parent topic, ϕ ϵ ′ , to draw a relative stick length for its child topic, ψ ϵ , creating a larger break if the stick to break is already short. as presented in appendix d, this new parametrization yields the average stick length e[ϕ|ℓ] ≈ 1/(2γ + 1/e[ϕ|ℓ-1]) for ℓ ≥ 2, which achieves an invariant partitioning scale by not decaying exponentially with ℓ. the solid lines in figure 3 depict the effect of this new parametrization. figure 2 (b) shows our scale-invariant tssb with the same hyperparameters as in (a). the probability of the topics is less likely to decrease at the deeper levels in (b). ## scale-invariant infinite hierarchical topic model we employ our scale-invariant tssb to model the hierarchical latent topics in a document. topic models consist of two types of distributions: documenttopic distributions for topic composition and topicword distributions for word emission. the ihlda leverages the scale-invariant tssb in section 3 to construct the former and employs a hierarchical pitman-yor process @xcite for the latter. combining both distributions will embed the topics into an infinite tree, which we call the ihlda, scale-invariant infinite hierarchical lda. ## experiments 6.1 data in our experiments, we used the bbc news corpus @xcite , the 20news corpus @xcite , and the original wikipedia corpus. the bbc news corpus contains 2,225 documents in five topic areas from the bbc news website, the 20news corpus is a collection of 18,828 posts from 20 usenet newsgroups, and the wikipedia corpus comprises 50,153 english articles randomly sampled from ten main categories foot_2 and their subcategories. we selected 80% of the data randomly for training. ## related work the existing models have heuristically addressed the issue of topic fragmentation by truncating topics at a certain threshold and truncating the tree structure to a small number of levels. @xcite constructed a hierarchical document model with the tssb foot_5 and truncated the topics with less than 50 assigned documents when the corpus had 1740 documents. the ntsntm @xcite sequentially selected the topics until the sum of probabilities in the corpus exceeded 95%. existing probabilistic approaches @xcite @xcite only consider three levels. @xcite introduced neural architectures but fixed the number of levels to three with an initial number of three branches for both the second and third levels. another advantage of the ihlda is that it employs a hierarchical bayesian extension of the tssb to draw a child tssb from the base tssb (see figure 4 ). unlike some probabilistic models that restrict a document-topic distribution to a single or multiple topic-path on a tree @xcite @xcite , ihlda does not limit topics that can appear in a document. hierarchical topic models have a wide range of extensions @xcite @xcite @xcite , and the ihlda is orthogonal to them and useful for these extensions. ## conclusion existing hierarchical topic models yield topics with exponentially smaller probabilities. to address this intrinsic issue, we propose the ihlda, a nonparametric bayesian model that learns a latent topic hierarchy with arbitrary depth and width. our model adjusts topic creation to achieve the expected topic probability without dependence on its depth, which can also improve other models that use the stickbreaking process. as a topic model, the ihlda is a hierarchical extension of the tssb and draws topic assignments efficiently without enumerating all possible candidates. our experiments on standard document datasets confirm that the ihlda outperforms the existing methods, including the latest neural models, and extracts meaningful topic structures with better hierarchical diversity and uniqueness.