Neural Mixed Counting Models for Dispersed Topic Discovery (2020)

Abstract: Mixed counting models that use the negative binomial distribution as the prior can well model over-dispersed and hierarchically dependent random variables; thus they have attracted much attention in mining dispersed document topics. However, the existing parameter inference method like Monte Carlo sampling is quite time-consuming. In this paper, we propose two efficient neural mixed counting models, i.e., the Negative Binomial-Neural Topic Model (NB-NTM) and the Gamma Negative Binomial-Neural Topic Model (GNB-NTM) for dispersed topic discovery. Neural variational inference algorithms are developed to infer model parameters by using the reparameterization of Gamma distribution and the Gaussian approximation of Poisson distribution. Experiments on real-world datasets indicate that our models outperform state-of-the-art baseline models in terms of perplexity and topic coherence. The results also validate that both NB-NTM and GNB-NTM can produce explainable intermediate variables by generating dispersed proportions of document topics.

## introduction mixture modeling is an essential topic in statistics and machine learning areas, owing to generating the random probability measure of data samples belonging to multiple clusters. in unsupervised learning tasks such as topic discovery, mixture modeling has gained increasing attention from researchers @xcite @xcite @xcite . specifically, mixture modeling over document words devotes to assign these words to different topics via random probability measures. hierarchical dirichlet process (hdp) @xcite is one of the representative methods in mixture modeling, which can characterize the two-level dependency of random probability measures. although we can use monte carlo sampling or variational inference to estimate the parameters in hdp, it requires the help of indirect construction of random variables such as the chinese restaurant franchise @xcite or the stick-breaking construction @xcite due to the lack of conjugation between the two-tier dirichlet processes. this makes the inference of hdp mostly complicated @xcite . the mixed counting models represented by the negative binomial (nb) process @xcite and the gamma negative binomial (gnb) process @xcite have solved this problem to a certain extent, in which, the normalized gnb process has been proven to be equivalent to hdp @xcite . because both nb and gnb processes satisfy the properties of completely random measures @xcite , the generative process of random probability measures among various mixed components is independent and becomes straightforward. moreover, they naturally introduce non-negative constraints and have been proven as able to model over-dispersed data. in the case of mining latent topics of documents, the over-dispersed property indicates that the variance is larger than the mean for document-topic distributions. when compared to the nb process, the gnb process has an extra feature of describing more flexible stochastic phenomena with hierarchical dependencies. despite the above advantages, with the increase of data size and observable information, the aforementioned parameter inference method like monte carlo sampling or variational inference has gradually become an important factor limiting the usage scenarios of mixed counting models @xcite . the reason is that monte carlo sampling has a high computational cost, and variational inference becomes intractable when applied to models with complex variable @xcite . neural variational inference (nvi) is a flexible and fast parameter inference framework based on neural networks @xcite . it can be regarded as a generalization of variational autoencoder applicable to natural language processing tasks. based on nvi, several neural topic models had been proposed and achieved encouraging performance in document modeling @xcite @xcite . these models used the neural network to learn the distribution relationship between input documents and latent topics due to its excellent function fitting ability and scalability. particularly, the neural network parameters can be trained by back-propagation through the reparameterization of a continuous distribution @xcite or using variance reduction techniques for a discrete distribution @xcite . however, the hidden variables in the above neural topic models lack good interpretability, and it is also impossible to model over-dispersed and hierarchically dependent document sets for these methods. in this paper, we propose two novel neural mixed counting models dubbed the negative binomial-neural topic model (nb-ntm) and the gamma negative binomial-neural topic model (gnb-ntm) based on nb and gnb processes, respectively. the general motivation is to combine the advantages of nvi and mixed counting models. on the one hand, nvi-based models are fast and easy to estimate but hard to interpret. on the other hand, document modeling via mixed counting models is easy to interpret but difficult to infer. in our nb-ntm and gnb-ntm, we develop nvi algorithms to infer parameters by using the reparameterization of gamma distribution and the gaussian approximation of poisson distribution. extensive experiments on real-world datasets validate the effectiveness of our proposed models in perplexity, topic coherence, and dispersed topic learning. furthermore, the proposed models can describe the hierarchical dependence of random probability measures and introduce non-negative constraints, which renders the intermediate variables generated by our methods to have good interpretability. the remainder of this article is organized as follows. in section 2, we summarize the related studies on topic discovery. in section 3, we introduce the definitions and properties of background methods. the proposed models are described in section 4, the experimental evaluations are shown in section 5, and we draw the conclusions in section 6. ## related work topic discovery aims to use the statistical information of word occurrences to obtain the abstract semantic structure embedded in a document set. from bayesian methods represented by latent semantic analysis (lsa) @xcite , probabilistic latent semantic analysis (plsa) (hofmann, 1999), latent dirichlet allocation (lda) @xcite , and hierarchical dirichlet process (hdp) @xcite , topic discovery had been widely researched in natural language processing and applied to many scenarios. for instance, the above models were extended to capture topic relevance @xcite and topic evolution over time @xcite . algorithms for short text @xcite , tagged data @xcite , and stream data @xcite were also proposed. considering the importance of prior distributions in lda-based models, some research efforts tried to use beta and gaussian distributions instead of the dirichlet distribution as the prior of probabilistic graphical models @xcite . although the bayesian method is a natural way to represent the latent structure of a document set in topic discovery, as the structure of such a model becomes deeper and more complex, pure bayesian inference becomes intractable due to the high dimensional integrals required @xcite . to address this issue, @xcite proposed a parallel monte carlo sampling method for hdp based on multi-threading. unfortunately, it needs to traverse every word of all topics (i.e., threads) in the whole corpus when updating the topic-word distribution, rendering a large time cost for thread communication. with the development of deep learning, especially the introduction of nvi, there is a new direction to discover topics based on neural networks. for example, @xcite assumed that word distributions in each document could be represented by hidden variables sampled from multiple gaussian distributions, and they used the variational lower bound as the objective function of their model named nvdm. @xcite employed the logical gaussian distribution to approxi-mate the dirichlet distribution, which improved the variational auto-encoder and lda simultaneously. @xcite proposed a method named gsm to model the document-topic distribution explicitly. in their study, the topic-word distribution was introduced into the decoder. besides the above nvibased methods, @xcite developed a stick-breaking variational auto-encoder for image generation. @xcite al. (2019 ) proposed a model named w-lda in the wasserstein autoencoder framework. they employed the maximum mean discrepancy (mmd) in w-lda to match the proposed distribution and the prior distribution. however, the accuracy of mmd relied heavily on the number of samples for each distribution, and the kernel function in mmd had a significant influence on the performance. by leveraging word embeddings, gupta et al. (2019) proposed a neural autoregressive topic model dubbed idocnade to enrich the context of short text. experiments indicate that idocnade outperformed state-of-the-art generative topic models. the recent relevant work to ours is the method proposed in @xcite , which regarded the nb distribution as the prior in modeling the over-dispersed discrete data. however, the parameters of this method were still derived from the latent variables that obey the gaussian distribution. thus, these latent variables do not satisfy the non-negative constraint and lack good interpretability. furthermore, the above method did not model topics explicitly, making it hard to generate document-topic and topic-word distributions. ## proposed models in this section, we respectively detail our nb-ntm and gnb-ntm for dispersed topic discovery. ## conclusion in this paper, we present two neural mixed counting models named nb-ntm and gnb-ntm. different from the current time consuming bayesian methods, our models apply to large-scale datasets through the efficient back-propagation algorithm and gpu acceleration. when compared to the existing neural topic models, both nb-ntm and gnb-ntm can well model the random variables with over-dispersed and hierarchically dependent characteristics. extensive experiments on real-world datasets validate the effectiveness of our models in terms of perplexity, topic coherence, and producing explainable intermediate variables by generating dispersed proportions of document topics. the results also indicate that nb distribution families can characterize text data aptly, which is essentially due to their conformity with the over-dispersed and sparse properties of natural language.