T ree- S tructured N eural T opic M odel (2020)

Abstract: This paper presents a tree-structured neural topic model, which has a topic distribution over a tree with an infinite number of branches. Our model parameterizes an unbounded ancestral and fraternal topic distribution by applying doubly-recurrent neural networks. With the help of autoencoding variational Bayes, our model improves data scalability and achieves competitive performance when inducing latent topics and tree structures, as compared to a prior tree-structured topic model (Blei et al., 2010). This work extends the tree-structured topic model such that it can be incorporated with neural models for downstream tasks.

## introduction probabilistic topic models, such as latent dirichlet allocation (lda; @xcite , are applied to numerous tasks including document modeling and information retrieval. recently, @xcite ; @xcite have applied the autoencoding variational bayes (aevb; @xcite framework to basic topic models such as lda. aevb improves data scalability in conventional models. the limitation of the basic topic models is that they induce topics as flat structures, not organizing them into coherent groups or hierarchies. treestructured topic models @xcite , which detect the latent tree structure of topics, can overcome this limitation. these models induce a tree with an infinite number of nodes and assign a generic topic to the root and more detailed topics to the leaf nodes. in figure 1 , we show an example of topics induced by our model. such characteristics are preferable for several downstream tasks, such as document retrieval @xcite , aspect-based sentiment analysis @xcite and extractive summarization (celikyilmaz root carry purchase cover 1: quality months zipper time back 11: sleeve inside inch protection nice 111: bottom cover top plastic scratches 112: color cover mac keyboard love 12: perfect quality price bought size 121: item return receive amazon money 122: price recommend buy perfect love 13: pockets carry strap shoulder compartment 131: big laptops tablet description hp 132: books school carry bags back and hakkani-tur, 2010), because they provide succinct information from multiple viewpoints. for instance, in the case of document retrieval of product reviews, some users are interested in the general opinions about bag covers, while others pay more attention to specific topics such as the hardness or color of the covers. the tree structure can navigate users to the documents with desirable granularity. however, it is difficult to use tree-structured topic models with neural models for downstream tasks. while neural models require a large amount of data for training, conventional inference algorithms, such as collapsed gibbs sampling @xcite or mean-field approximation @xcite , have data scalability issues. it is also desirable to optimize the tree structure for downstream tasks by jointly updating the neural model parameters and posteriors of a topic model. to overcome these challenges, we propose a treestructured neural topic model (tsntm), which is parameterized by neural networks and is trained using aevb. while prior works have applied aevb to flat topic models, it is not straightforward to parameterize the unbounded ancestral and fraternal topic distribution. in this paper, we provide a solution to this by applying doubly-recurrent neural networks (drnn; @xcite , which have two recurrent structures over respectively the ancestors and siblings. experimental results show that the tsntm achieves competitive performance against a prior work @xcite when inducing latent topics and tree structures. the tsntm scales to larger datasets and allows for end-to-end training with neural models of several tasks such as aspect-based sentiment analysis @xcite and abstractive summarization @xcite . ## related works following the pioneering work of tree-structured topic models by @xcite , several extended models have been proposed @xcite @xcite @xcite . our model is based on the modeling assumption of @xcite ; @xcite , while parameterizing a topic distribution with aevb. in the context of applying aevb to flat document or topic modeling @xcite @xcite , @xcite proposed a model, which is closely related to ours, by applying recurrent neural networks (rnn) to parameterize an unbounded flat topic distribution. our work infers the topic distributions over an infinite tree with a drnn, which enables us to induce latent tree structures. @xcite used a tree-structured topic model @xcite with a variational autoencoder (vae) to represent video frames as a tree. however, their approach is limited to smaller datasets. in fact, they used only 1,241 videos (corresponding to documents) for training and separately updated the vae parameters and the posteriors of the topic model by mean-field approximation. this motivates us to propose the tsntm, which scales to larger datasets and allows for end-to-end training with neural models for downstream tasks. ## tree-structured neural topic model we present the generative process of documents and the posterior inference by our model. as shown in figure 2 , we draw a path from the root to a leaf node and a level for each word. the word is drawn from the multinomial distribution assigned to the topic specified by the path and level. 1. for each document index @xmath0, . . . , d}: draw a gaussian vector: obtain a level distribution: 2. for each word index @xmath1, . . . , n d } in d: draw a word: here, @xmath2, . . . , k} and par(k) denote the k-th topic and its parent, respectively. @xmath3, . . . , l} denotes the l-th level. see appendix a.1 for more details. in contrast, we introduce neural architectures, f π and f θ , to transform a gaussian sample to a topic distribution, allowing for posterior inference with aevb. specifically, we apply a drnn to parameterize the path distribution over the tree. ## conclusion we proposed a novel tree-structured topic model, the tsntm, which parameterizes the topic distribution over an infinite tree by a drnn. experimental results demonstrated that the tsntm achieves competitive performance when inducing latent topics and their tree structures, as compared to a prior tree-structured topic model @xcite . with the help of aevb, the tsntm can be trained approximately 15 times faster and scales to larger datasets than the ncrpbased model. this allows the tree-structured topic model to be incorporated with recent neural models for downstream tasks, such as aspect-based sentiment analysis @xcite and abstractive summarization @xcite . by incorporating our model instead of flat topic models, they can provide multiple information with desirable granularity.