Neural Topic Modeling with Bidirectional Adversarial Training (2020)

Abstract: Recent years have witnessed a surge of interests of using neural topic models for automatic topic extraction from text, since they avoid the complicated mathematical derivations for model inference as in traditional topic models such as Latent Dirichlet Allocation (LDA). However, these models either typically assume improper prior (e.g. Gaussian or Logistic Normal) over latent topic space or could not infer topic distribution for a given document. To address these limitations, we propose a neural topic modeling approach, called Bidirectional Adversarial Topic (BAT) model, which represents the first attempt of applying bidirectional adversarial training for neural topic modeling. The proposed BAT builds a two-way projection between the document-topic distribution and the document-word distribution. It uses a generator to capture the semantic patterns from texts and an encoder for topic inference. Furthermore, to incorporate word relatedness information, the Bidirectional Adversarial Topic model with Gaussian (Gaussian-BAT) is extended from BAT. To verify the effectiveness of BAT and Gaussian-BAT, three benchmark corpora are used in our experiments. The experimental results show that BAT and Gaussian-BAT obtain more coherent topics, outperforming several competitive baselines. Moreover, when performing text clustering based on the extracted topics, our models outperform all the baselines, with more significant improvements achieved by Gaussian-BAT where an increase of near 6% is observed in accuracy.

## introduction topic models have been extensively explored in the natural language processing (nlp) community for unsupervised knowledge discovery. latent dirichlet allocation (lda) @xcite , the * corresponding author most popular topic model, has been extended @xcite @xcite for various extraction tasks. due to the difficulty of exact inference, most lda variants require approximate inference methods, such as mean-field methods and collapsed gibbs sampling. however, these approximate approaches have the drawback that small changes to the modeling assumptions result in a re-derivation of the inference algorithm, which can be mathematically arduous. one possible way in addressing this limitation is through neural topic models which employ blackbox inference mechanism with neural networks. inspired by variational autoencoder (vae) @xcite , @xcite used the logistic-normal prior to mimic the simplex in latent topic space and proposed the neural variational lda (nvlda). moreover, they replaced the word-level mixture in nvlda with a weighted product of experts and proposed the prodlda @xcite to further enhance the topic quality. although @xcite used the logistic-normal distribution to approximate the dirichlet distribution, they are not exactly the same. an illustration of these two distributions is shown in figure 1 in which the logistic-normal distribution does not exhibit multiple peaks at the vertices of the simplex as that in the dirichlet distribution and as such, it is less capable to capture the multi-modality which is crucial in topic modeling @xcite . to deal with the limitation, @xcite proposed the adversarialneural topic model (atm) based on adversarial training, it uses a generator network to capture the semantic patterns lying behind the documents. however, given a document, atm is not able to infer the document-topic distribution which is useful for downstream applications, such as text clustering. moreover, atm take the bag-of-words assumption and do not utilize any word relatedness information captured in word embeddings which have been proved to be crucial for better performance in many nlp tasks @xcite . to address these limitations, we model topics with dirichlet prior and propose a novel bidirectional adversarial topic model (bat) based on bidirectional adversarial training. the proposed bat employs a generator network to learn the projection function from randomly-sampled documenttopic distribution to document-word distribution. moreover, an encoder network is used to learn the inverse projection, transforming a document-word distribution into a document-topic distribution. different from traditional models that often resort to analytic approximations, bat employs a discriminator which aims to discriminate between real distribution pair and fake distribution pair, thereby helps the networks (generator and encoder) to learn the two-way projections better. during the adversarial training phase, the supervision signal provided by the discriminator will guide the generator to construct a more realistic document and thus better capture the semantic patterns in text. meanwhile, the encoder network is also guided to generate a more reasonable topic distribution conditioned on specific document-word distributions. finally, to incorporate the word relatedness information captured by word embeddings, we extend the bat by modeling each topic with a multivariate gaussian in the generator and propose the bidirectional adversarial topic model with gaussian (gaussian-bat). the main contributions of the paper are: our work is related to two lines of research, which are adversarial training and neural topic modeling. ## methodology our proposed neural topic models are based on bidirectional adversarial training @xcite and aim to learn the two-way non-linear projection between two high-dimensional distributions. in this section, we first introduce the bidirectional adversarial topic (bat) model that only employs the word co-occurrence information. then, built on bat, we model topics with multivariate gaussian in the generator of bat and propose the bidirectional adversarial topic model with gaussian (gaussian-bat), which naturally incorporates word relatedness information captured in word embeddings into modeling process. ## experiments in this section, we first present the experimental setup which includes the datasets used and the baselines, followed by the experimental results. ## conclusion in this paper, we have explored the use of bidirectional adversarial training in neural topic models and proposed two novel approaches: the bidirectional adversarial topic (bat) model and the bidirectional adversarial topic model with gaussian (gaussian-bat). bat models topics with the dirichlet prior and builds a two-way transformation between document-topic distribution and document-word distribution via bidirectional adversarial training. gaussian-bat extends from bat by incorporating word embeddings into the modeling process, thereby naturally considers the word relatedness information captured in word embeddings. the experimental comparison on three widely used benchmark text corpus with the existing neural topic models shows that bat and gaussian-bat achieve improved topic coherence results. in the future, we would like to devise a nonparametric neural topic model based on adversarial training. besides, developing correlated topic modelsis another promising direction.