Tree-Structured Topic Modeling with Nonparametric Neural Variational Inference (2021)

Abstract: Topic modeling has been widely used for discovering the latent semantic structure of documents, but most existing methods learn topics with a flat structure. Although probabilistic models can generate topic hierarchies by introducing nonparametric priors like Chinese restaurant process, such methods have data scalability issues. In this study, we develop a tree-structured topic model by leveraging nonparametric neural variational inference. Particularly, the latent components of the stick-breaking process are first learned for each document, then the affiliations of latent components are modeled by the dependency matrices between network layers. Utilizing this network structure, we can efficiently extract a tree-structured topic hierarchy with reasonable structure, low redundancy, and adaptable widths. Experiments on real-world datasets validate the effectiveness of our method.

## introduction topic models @xcite are important tools for discovering latent semantic patterns in a corpus. these models can be grouped into flat models and hierarchical models. in many domains, topics can be naturally organized into a tree, where the hierarchical relationships among topics are valuable for data analysis and exploration. tree-structured topic model @xcite was thus developed to learn coherent topics from text without disrupting the inherent hierarchical structure. such a method has been proven as useful in various downstream applications, including hierarchical categorization of web pages @xcite , aspects hierarchies extraction in reviews @xcite , and hierarchies discovery of research topics in academic repositories @xcite . despite the practical importance and potential advantages, tree-structured topic models still face the following challenges. firstly, the hierarchical structure of topics should be reasonable @xcite . typically, topics near the root are more general while the ones close to the leaves are more specific. besides, child topics should be coherent with their corresponding parent topics. secondly, low redundancy is necessary for the extracted topics, in order to prevent the distributions associated with parent topics and their children being extremely similar @xcite . thirdly, the number of topics in each hierarchy level should be automatically determined by the model, because it is usually unknown and can not be previously set to a predefined value @xcite . finally, it is difficult for probabilistic models to enhance the data scalability @xcite . previously, several tree-structured topic models @xcite @xcite have been developed. but these methods can not fully overcome the aforementioned challenges. in this paper, we focus on grouping topics into a reasonable tree structure, based on the neural variational inference (nvi) framework @xcite with a nonparametric prior. owing to the excellent function fitting ability, neural network has been widely introduced into topic modeling. nonetheless, few neural methods explicitly model the dependencies among different layers and get explainable hierarchical topics, which is largely due to the weak interpretability of neural networks. furthermore, the inflexibility of neural networks also makes it difficult to learn an unbounded number of topics at each level. to address these limitations, we propose a novel nonparametric neural method to generate tree-structured topic hierarchies, namely nonparametric tree-structured neural topic model (ntsntm) foot_0 . by connecting the network layers with dependency matrices, the model is able to extract an explainable tree-structured hierarchy. firstly, the topic affiliations among hierarchy levels can be determined by the dicrete vectors of the dependency matrices. secondly, to control redundancy among topics, we allow the model to freely generate topics without duplicating their corresponding parent topics. thirdly, we couple a stick-breaking process with nvi to equip the topic tree with self-determined widths, which can help the model determine the number of topics automatically. finally, due to the advantages of neural networks, our model can scale to larger datasets conveniently. experiments indicate that our model outperforms baselines on several widely adopted metrics and two new measurements developed for tree-structured topic models. the rest of this paper is organized as follows. we describe related work in section 2. then, we detail the proposed ntsntm in section 3. section 4 presents our experimental results and discussions. finally, we draw conclusions in section 5. ## related work in @xcite , a tree-structured topic model called hlda was first proposed by introducing a nested chinese restaurant process (ncrp). for hlda, a topic tree is constructed through gibbs sampling given a certain depth. based on hlda, @xcite proposed a knowledged-based htm to generate topic hierarchies from multiple domains corpora, but the hierarchical relation between the ancestor topic and the offspring one may be unclear, because a document is generated by the topics along a single path of the tree. to overcome this issue, @xcite proposed a recursive crp (rcrp), in which a document possesses a distribution over the entire tree. although rcrp has shown remarkable competitiveness in hierarchical topic modeling, it suffers from the major limitation of data scalability @xcite . several other methods focused on hierarchical text clustering. for instance, @xcite applied nested stick-breaking processes to cluster data into a tree structure. unfortunately, the above method only models a document by a single node of the tree. @xcite developed a model named hlta for topic detection, in which words and top-ics are clustered by employing the bridged-islands algorithm iteratively. however, hlta is unable to cope with polysemous words, which is quite important for topic models. to couple nonparametric processes with nvi, @xcite used gaussian distributions to generate stick-breaking fractions. @xcite first described how to use stochastic gradient variational bayes for posterior inference of the weights in stick-breaking processes. experiments indicated that the latent representations of the above model were more discriminative than those of the gaussian variant. then, @xcite developed two nonparametric neural topic models by treating topics as trainable parameters. unfortunately, the aforementioned methods can only learn topics with a flat structure. for tree-structured neural topic modeling, a feasible way is to decompose the distribution over the topic tree into a path distribution and a level distribution. following @xcite , where a tree-based stick-breaking construction of ncrp was first derived to draw topic paths, and then a level distribution was learned to sample topics along the path, @xcite proposed a tree-structured neural topic model (tsntm) by parameterizing an unbounded ancestral and fraternal topic distribution. tsntm applies a doublyrecurrent neural network (drnn) to obtain topic embeddings via ancestral and fraternal edges, then generates breaking fractions by the dot product between document embeddings and topic embeddings. however, tsntm fails to learn a reasonable topic tree for the following reasons. firstly, the breaking fractions do not obey the beta distributions adopted in the stick-breaking process (sbp). secondly, the structure of drnn in tsntm is simplified, where the topic embeddings are generated directly by an initialized root embedding and two parameter matrices (i.e., ancestral and fraternal connections). this prevents the model from learning appropriate semantic embeddings for topics. finally, tsntm relies on heuristic rules to update the tree structure. another stream of work is to generate a document by a directed acyclic graph (dag) structured topic hierarchy. for instance, @xcite introduced the pachinko allocation model (pam) to capture correlations between topics using a dag. @xcite proposed the hierarchical pam by connecting the root topic to lower-level topics through multinomial distributions. nonprobabilistic matrix factorization was also used to extract the topic structure. @xcite used non-negative matrix factorization (nmf) with three optimization constraints, including global independence, local independence, and information consistency, to preserve topic coherence and a reasonable structure. @xcite incorporated pre-trained word embeddings into nmf to further improve topic coherence. the main limitation of nmf-based methods, however, is that a time-consuming process (e.g., measure the stability of results by running multiple random samplings) is necessary to determine the number of topics at each level. this is because nonparametric priors are intractable to be included in these models. ## tree-structured neural topic model with nonparametric prior in this section, we firstly describe the stickbreaking process. then, we introduce the modeling of tree-structured topic hierarchy. finally, we detail the inference method of our ntsntm. ## conclusion in this paper, we propose a nonparametric treestructured neural topic model named ntsntm. our method explicitly models the dependency of latent variables from different layers, and combines them to reconstruct the input text. by coupling sbp with dependency matrices, we can update the tree structure automatically. extensive experiments validate the effectiveness of our ntsntm on generating a reasonable topic tree with low topic redundancies. furthermore, our model can be trained 2 times faster than the existing nvi-based tsntm with approximately 800k documents. in the future, we plan to apply our method to aspect extraction.