Nonlinear Structural Equation Model Guided G aussian Mixture Hierarchical Topic Modeling (2023)

Abstract: Hierarchical topic models, which can extract semantically meaningful topics from a textcorpus in an unsupervised manner and automatically organise them into a topic hierarchy, have been widely used to discover the underlying semantic structure of documents. However, the existing models often assume in the prior that the topic hierarchy is a tree structure, ignoring symmetrical dependenciesbetween topics at the same level. Moreover, the sparsity of text data often complicate the analysis. To address these issues, we propose NSEM-GMHTM as a deep topic model, witha Gaussian mixture prior distribution to improve the model’s ability to adapt to sparse data, which explicitly models hierarchical and symmetric relations between topics through the dependency matrices and nonlinear structural equations. Experiments on widely used datasets show that our NSEM-GMHTM generates more coherent topics and a more rational topic structure when compared to state-of-theart baselines. Our code is available at https: //github.com/nbnbhwyy/NSEM-GMHTM.

## introduction topic models, which can uncover the hidden semantic structure in a text corpus, have been widely applied to text analysis. specifically, a topic model aims to discover a set of semantically meaningful topics from a document set. each topic captures a common pattern of word co-occurrences in the document and is often interpreted semantically as a coherent set of words representing a common concept. although traditional topic models like the latent dirichlet allocation (lda) @xcite and the embedded topic model (etm) @xcite are able to achieve this goal, they assume that topics are independent, which limits the ability of these models to explore the topic structure. to remedy the defect, a series of hierarchical extensions, such as the hierarchical lda (hlda) @xcite , the recursive chinese restaurant process (rcrp) @xcite , and the nested hierarchical dirichlet process (nhdp) @xcite , have been proposed. commonly, these models learn hierarchical topics in tree structures, which assume that the topics in the upper layers are more general/abstract than those in the lower layers. consequently, revealing hierarchical relations between topics provides the user an intuitive way to better understand text data. however, these methods rely on approximate approaches (e.g., variational inference and gibbs sampling) and require complex derivation or high computational costs to estimate parameters. with the development of deep neural networks and the proposal of neural variational inference (nvi), there is a growing interest in developing neural hierarchical topic models (nhtms) due to their fast parameter inference and flexibility @xcite @xcite . generally, nhtms are based on variational auto-encoder (vae) and model topic hierarchy as the relationship between neurons at different levels in the encoder or decoder, such as the tree-structured neural topic model (tsntm) @xcite and the nonparametric tsntm (ntsntm) @xcite . however, most nhtms rely on a single isotropic multivariable gaussian prior distribution, which often fails to well approximate the posterior distributions of sparse data @xcite . a tighter estimation of the posterior distribution could greatly improve the power of vae in fitting and analyzing sparse data. moving beyond topic mining, the application of gaussian mixture model (gmm) as a priori for latent variables has recently shown promising performance in the fields of image generation and bioinformatics @xcite . we further note that previous nhtms have focused only on relationships between topics at different levels. for example, @xcite built a topic tree bottom-up through a dependency matrix, where the parent topic can be considered as a generalization of its child topics. however, the generation of high level topics may also be influenced by the structure between topics at lower levels. for example, in the case of modules in biochemical networks or communities in social networks, information cross-talk between nodes at the same level plays a crucial role in the extraction of higher level abstraction modules @xcite . furthermore, returning to the nature of topics, they can be thought of as words with highly generalised semantics. intuitively, as defined by @xcite , not only are there hierarchical relations between topics with different levels of generalisation like chicago and city (isa), but there should also be symmetrical relations that belong to the same level, such as cut and knife (capableof) and learned and learned (relatedto). unfortunately, existing nhtms tend to predefine topics as tree structures, focusing only on modelling topic hierarchy relationships and neglecting symmetrical relationships between topics that may also help researchers better understand and process textual information. furthermore, the use of topic symmetric structures to help models better capture document semantics has not been much explored. in addition, some works @xcite generate a document via directed acyclic graph (dag) structured topics, but the structure of their generated topics is often unclear. to overcome these limitations, in this paper, we propose the nonlinear structural equation model guided gaussian mixture hierarchical topic model (nsem-gmhtm), a deep generative model of documents. as shown in figure 1 , in contrast to the previous hierarchical topic models, the core idea is to apply a nonlinear structural equation model (nsem) to explicitly construct the symmetric dependencies between topics to facilitate the extraction of a more comprehensive and clear topic structure. in particular, we introduce gaussian mixture distribution as a prior for latent variables, enabling the network to learn more complex distributions, and further improving the power of vae in fitting and analyzing sparse data. experiments show that our model outperforms state-of-the-art baselines on several widely adopted metrics, validating the rationality of topic structure generated by our model. futhermore, ablation studies and extensive qualitative evaluations have shown that nsem guided nhtm results in a better topic structure, which further demonstrates the validity of our method. ## related work following the pioneering work on topic models @xcite , several extension models, such as hlda @xcite and rcrp @xcite have been proposed to explore the relationships between topics. although these models showed clear competitiveness in hierarchical topic modeling, they are limited by the expensive iterative inference step, which is not conducive to further model expansion @xcite . nvi-based topic models @xcite @xcite commonly converted a document to a bag-of-words (bow) representation determined on the frequency count of each vocabulary token in the document. the bow input was processed through an mlp followed by variational inference which sampled a latent document-topic vector. a decoder network then reconstructed the original bow using the latent document-topic vector via a topic-word distribution. building hierarchical topic models based on nvi is a promising direction due to the fast parameter inference and flexibility. @xcite proposed a tree-structured neural topic model, which applied doubly-recurrent neural networks to parameterize topic distributions over a tree. @xcite developed a tree-structured topic model by using nonparametric nvi, which first learned the potential components of the stickbreaking process for each document and then modelled the affiliation of components through depen-dency matrices between network layers. besides tree-structured topic models, several works proposed to generate a document by a dag structured topic hierarchy. for instance, @xcite proposed the hierarchical pachinko allocation model (pam) by connecting the root topic to lower-level topics through multinomial distributions. @xcite and @xcite applied nonnegative matrix factorization (nmf) to generate hierarchical topics in a dag structure. although the aforementioned dag-based approaches captured all the relations between topics, the generated topic structures of them were not clear compared to those of the tree-structured topic models. in turn, the tree-structured topic models ignored the symmetric relations between topics at the same level. unlike previous approaches, our approach uses nsem and dependency matrices to capture both symmetric and hierarchical dependencies between topics, which helps to further clarify the structure of topics. recently, some works attempted to use other prior distributions. for neural topic models, @xcite combined the mixed counting models and variational inference to develop the negative binomial neural topic model (nb-ntm) and the gamma negative binomial neural topic model (gnb-ntm). for hntms, @xcite proposed sawetm, which used a weibull prior to model sparse and nonnegative documents, and mitigated the problem of posterior collapse to some extent with a sawtooth connection module. @xcite built on an existing method @xcite by proposing to embed topics and words into a hyperbolic space, which enhanced the model's ability to mine the implicit semantic hierarchy. for a more comprehensive comparison of the models, we used these models as baselines for our work. ## the proposed model in this section, we propose nsem-gmhtm for text analysis, which aims at exploring a topic structure. the motivation for designing nsem-gmhtm focuses on tackling two main challenges: (i) how to clearly construct topic symmetric and hierarchical dependencies; (ii) how to design expressive neural networks to improve the ability of models to adapt and analyse sparse data. below, we firstly introduce the details of the related technology, and then describe the decoder and encoder of nsem-gmhtm as shown in figure 2 . finally, we provide details of model inference. ## conclusion in this paper, we propose a novel neural topic model named nsem-gmhtm. our method explicitly constructs symmetric and hierarchical dependencies between topics through nsem and dependency matrices. in addition, we introduce gmm as a prior for latent variables to improve the ability of nsem-gmhtm to fit and analyze sparse data. extensive experiments have shown that our method outperforms state-of-the-art baselines in extracting coherent and reasonably structured topics. furthermore, with learned word and topic embeddings, and different types of topic relationships (hierarchical and symmetric), nsem-gmhtm can discover a clearly interpretable topic structure. eventually, the topic structures mined by nsem-gmhtm show defined topic associations beyond the hierarchy, which are more consistent with the semantic relations of generic knowledge graphs such as wordnet @xcite and conceptnet @xcite compared to other nhtms, suggesting that our model may be able to exploit knowledge more fully. in the future, we will attempt to further incorporate prior information to guide the discovery of topic structures. in summary, our findings suggest that the discovery of topic structure can benefit from the construction of topic symmetric relations, which may contribute to a better understanding of text data.