Neural Topic Modeling with Cycle-Consistent Adversarial Training (2020)

Abstract: Advances on deep generative models have attracted significant research interest in neural topic modeling. The recently proposed Adversarial-neural Topic Model models topics with an adversarially trained generator network and employs Dirichlet prior to capture the semantic patterns in latent topics. It is effective in discovering coherent topics but unable to infer topic distributions for given documents or utilize available document labels. To overcome such limitations, we propose Topic Modeling with Cycle-consistent Adversarial Training (ToMCAT) and its supervised version sToMCAT. ToMCAT employs a generator network to interpret topics and an encoder network to infer document topics. Adversarial training and cycle-consistent constraints are used to encourage the generator and the encoder to produce realistic samples that coordinate with each other. sToMCAT extends ToMCAT by incorporating document labels into the topic modeling process to help discover more coherent topics. The effectiveness of the proposed models is evaluated on unsupervised/supervised topic modeling and text classification. The experimental results show that our models can produce both coherent and informative topics, outperforming a number of competitive baselines.

## introduction topic models, such as latent dirichlet allocation (lda) @xcite , aim to discover underlying topics and semantic structures from text collections. due to its interpretability and effectiveness, lda has been extended to many natural language processing (nlp) tasks @xcite @xcite . most of these models employ mean-field variational inference or collapsed gibbs sampling @xcite for model inference as a result of their intractable posteriors. however, such inference algorithms are model specific and require dedicated derivations. to address such limitation, neural topic models with black-box inference have been explored, with more flexible training schemes. inspired by variational autoencoder (vae) @xcite , @xcite proposed neural variational document model which interprets the latent code in vae as topics. following this way, srivastava and sutton (2017) adopted the logistic normal prior rather than gaussian to mimic the simplex properties of topic distribution. logistic normal is a laplace approximation to the dirichlet distribution @xcite . however, logistic normal can not exhibit multiple peaks at the vertices of the simplex as the dirichlet distribution. therefore, it is less capable of capturing the multi-modality which is crucial for topic modeling @xcite . to overcome such limitation, @xcite proposed adversarial-neural topic model (atm), a topic model based on generative adversarial networks (gans) @xcite and sampling topics directly from the dirichlet distribution to impose a dirichlet prior. atm employs a generator transforming randomly sampled topic distributions to word distributions, and an adversarially trained discriminator estimating the probability that a word distribution came from the training data rather than the generator. although atm was shown to be effective in discovering coherent topics, it can not be used to induce the topic distribution given a document due to the absence of a topic inference module. such limitation hinders its application to downstream tasks, such as text classification. moreover, atm fails to deal with document labels which can help extract more co-herent topics. for example, a document labeled as 'sports' more likely belongs to topics such as 'basketball' or 'football' rather than 'economics' or 'politics'. to address such limitations of atm, we propose a novel neural topic modeling approach, named topic modeling with cycle-consistent adversarial training (tomcat). in tomcat, topic modeling is cast into the transformation between topic distributions and word distributions. specifically, the transformation from topic distributions to word distributions is used to interpret topics, and the reverse transformation is used to infer underlying topics for a given document. under such formulation, tomcat employs a generator to transform topic distributions randomly sampled from the dirichlet prior into the corresponding word distributions, and an encoder to reversely transform documents represented as word distributions into their topic distributions. to encourage the generator/encoder to produce more realistic target samples, discriminators for word/topic distributions are introduced to enable adversarial training. additional cycleconsistency constraints are utilized to align the learning of the encoder and the generator to prevent them from contradicting each other. furthermore, for documents with labels, we propose stomcat that introduces an extra classifier to regularize the topic modeling process. the main contributions of the paper are: our work is related to neural topic modeling and unsupervised style transfer. ## methodology given a corpus d consisting of n documents {x i } n @xmath0, two main purposes of topic modeling are: 1. topic discovery. given a one-hot topic indicating vector i @xmath1, discover the corresponding word distribution t @xmath2. more generally, we can consider topic discovery as finding a mapping from topic distribution to word distribution. we now formalize the above observations. let x be the word distribution set and z the topic distribution set. given training samples {x i } n @xmath3, the goal of topic modeling is to learn a mapping function g, called generator, to transform samples in z into x and a reverse function e, called encoder, to transform samples in x into z. however, it should be noted that training samples in x and z are unpaired since the topic distribution of a document is unknown before topic modeling. thus, the problem is how to learn g and e to model topics in the absence of paired samples between x and z. ## experiments in this section, we first describe datasets and compared baselines. ## conclusion we have presented tomcat, a neural topic model with adversarial and cycle-consistent objectives, and its supervised extension, stomcat. tomcat employs a generator to capture semantic patterns in topics and an encoder to encode documents into their corresponding topics. stomcat further incorporates document labels into topic modeling. the effectiveness of tomcat and stomcat is verified by experiments on topic modeling and text classification. in the future, we plan to extend our model to cope with external word or document semantics. it would also be interesting to explore alternative architectures other than cyclegan under our formulation of topic modeling. deyu zhou, xuan zhang, and yulan he. 2017. event extraction from twitter using non-parametric bayesian mixture model with word embeddings. in proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 1, long papers, pages 808-817, valencia, spain. association for computational linguistics. jun-yan zhu, taesung park, phillip isola, and alexei a. efros. 2017. unpaired image-to-image translation using cycle-consistent adversarial networks. in 2017 ieee international conference on computer vision (iccv). ieee.