Nonparametric Forest-Structured Neural Topic Modeling (2022)

Abstract: Neural topic models have been widely used in discovering the latent semantics from a corpus. Recently, there are several researches on hierarchical neural topic models since the relationships among topics are valuable for data analysis and exploration. However, the existing hierarchical neural topic models are limited to generate a single topic tree. In this study, we present a nonparametric forest-structured neural topic model by firstly applying the self-attention mechanism to capture parent-child topic relationships, and then build a sparse directed acyclic graph to form a topic forest. Experiments indicate that our model can automatically learn a forest-structured topic hierarchy with indefinite numbers of trees and leaves, and significantly outperforms the baseline models on topic hierarchical rationality and affinity.

## introduction topic model has been widely used in modeling a collection of documents and encoding the text content to a low dimensional feature space. traditional topic models can be divided into probabilistic graphical models and matrix factorization based methods. probabilistic graphical models, such as latent dirichlet allocation (lda) @xcite , rely on approximate approaches (e.g., variational inference and gibbs sampling) with complex derivation or high computational costs to estimate parameters @xcite . matrix factorization based methods @xcite can effectively decompose the documentword representation into two sub-matrices but are subject to a low stability @xcite . recently, neural topic models based on neural variational inference (nvi) @xcite @xcite have attracted great attention owing to the advantages of fast parameter inference and flexibility. the forest-structured topic models still face challenges since the hierarchical structure of topics needs to be (1) rational-root topics are general and children topics are specific to their corresponding parent topics @xcite ; (2) affinitiveeach topic is more similar to their children topics than topics from other parents @xcite ; in this work, we propose a nonparametric foreststructured neural topic model (nfntm) to tackle these challenges, which firstly captures parentchild topic relationships based on the self-attention mechanism @xcite foot_0 , and then learns a sparse directed acyclic graph (dag) to build a topic forest by federating document-topic distributions and parent-child relationships based on nvi. to our best knowledge, the current topic models with a dag structure are based on bayesian learning @xcite or non-negative matrix factorization (nmf) @xcite , and there is no work under the nvi framework. to sum up, the main contributions are summarized as follows: ## related work traditional topic models, such as lda @xcite , are powerful tools for modeling text in an unsupervised fashion, while they lack the exploration of the relationship among topics. to overcome this issue, a tree-structured topic model named hlda @xcite was first proposed. in hlda, the nested chinese restaurant process (ncrp) was used to generate a topic tree. to alleviate the single-path constraint assumed by ncrp, a nested hierarchical dirichlet process (hdp), i.e., nhdp @xcite was developed, which provided the ability of cross-thematic borrowing while keeping general topic areas in separate subtrees. the nested chinese restaurant franchise (ncrf) process developed in @xcite combined the advantages of hdp @xcite and ncrp. in @xcite , the recursive chinese restaurant process (rcrp) was proposed to discover a hierarchical topic structure with unbounded depth and width. these models based on the chinese restaurant process can be effectively employed to discover the hierarchical topic structure by bayesian learning, but the posterior inference method requires a high computational cost. the scalability of nmfbased methods @xcite is also quite limited. there is a new direction to build tree-structured topic models based on nvi due to its advantages of fast parameter inference and flexibility. a tree-structured neural topic model (tsntm) @xcite was proposed, which applied doubly-recurrent neural networks to parameterize topic distributions over a tree. but it lacked the ability of learning appropriate semantic embeddings for topics and relied on heuristic rules to update the tree structure. a nonparametric treestructured neural topic model (ntsntm) @xcite tackled these weaknesses by directly sampling the leaf topics and generating the paths from bottom up automatically. ntsntm used a common stick-breaking construction to infer topic distributions from the leaf nodes to the root node and applied dependency matrices to keep track of the affiliations among topics. however, the dependency matrices which determine the topic hierarchy are neural weights between the network layers. it results in the structure (i.e., depth and width) of the tree can only be set in advance. ## conclusion in this work, we present a forest-structured neural topic model named nfntm. particularly, the self-attention mechanism is applied to capture topic correlation and an augmented lagrangian method is employed to build the topic forest under dag constraints. by computing reparameterization of gamma distribution, the sbp is adopted to obtain a nonparametric model. we empirically demonstrate that our nfntm generates more rational, affinitive, and diverse topics than state-of-the-art hierarchical topic models, meanwhile achieves better topic interpretability than various baselines. although flat topic models can generate distinguishable topics with larger npmi scores than hierarchical ones, they treat all topics independently without explaining the corpus well to a user. party government support right general released release version original feature team won final championship sport match championship wrestling title wwe territory land power led empire engine power designed machine equipment food restaurant meat mushroom chicken disease brain treatment patient effect don texas frank joe didn point career record tech michigan god poem belief religion divine september august october june december blue flag background green light family father son life child test match england wicket cricket idea reason suggested influence stating long large small great le character scene episode story plot received award gave u club according including stated did making run inning league series baseball united state american national country second race lead point team gun inch class battery battleship nhl hockey named montreal stanley force attack air weapon bomb division general north corp line officer commander soldier war service indian india temple hindu indie tower castle temple stone museum british captain crew battle ordered second place end just week force army battle attack war police prison murder file crime film produc french france saint la louis greek al byzantine egypt ottoman state office house federal district modern history period followed century metal element chemical energy nuclear g henry england royal queen ha million estimated development approximately episode series season television rating mexico flooding florida issued gulf south west north east western day hour people home left bridge foot station mile built population area century largest national japanese japan s despite tokyo christmas discus holiday tom expectation early began continued despite took book novel story published comic world age open war international court law case right act fleet ship ton squadron admiral air aircraft squadron fighter flying water sea land port supply used like form body cause school student campus class high homer bart lisa jones marge court said case trial told design use used level unit building century built wall site australian australia zealand victoria sydney battalion brigade regiment infantry division main instead rest provided included canada canadian ice toronto hockey half victory took result came work book life century written damage area heavy rain caused world event tournament cup champion figure 11: visualization of the topic forest generated from wikitext-103. possibly due to a large variety of topics in this corpus, the result presents a more dispersed and shallower structure than others. the top words of two root topics and their children topics are (1) [day, hour, people, home, left]-→[court, law, case, right, act] and [party, government, support, right, general]; (2) [main, instead, rest, provides, included]-→[christmas, discus, holiday, tom, expectation] and [gun, inch, class, battery, battleship].