Neural Topic Modeling based on Cycle Adversarial Training and Contrastive Learning (2023)

Abstract: Neural topic models have been widely used to extract common topics across documents. Recently, contrastive learning has been applied to variational autoencoder-based neural topic models, achieving promising results. However, due to the limitation of the unidirectional structure of the variational autoencoder, the encoder is enhanced with the contrastive loss instead of the decoder, leading to a gap between model training and evaluation. To address the limitation, we propose a novel neural topic modeling framework based on cycle adversarial training and contrastive learning to apply contrastive learning on the generator directly. Specifically, a self-supervised contrastive loss is proposed to make the generator capture similar topic information, which leads to better topic-word distributions. Meanwhile, a discriminative contrastive loss is proposed to cooperate with the self-supervised contrastive loss to balance the generation and discrimination. Moreover, based on the reconstruction ability of the cycle generative adversarial network, a novel data augmentation strategy is designed and applied to the topic distribution directly. Experiments have been conducted on four benchmark datasets and results show that the proposed approach outperforms competitive baselines.

## introduction topic modeling, uncovering the semantic structures within a collection of documents, has been widely used in various natural language processing (nlp) tasks @xcite @xcite @xcite . latent dirichlet allocation (lda) @xcite , a probabilistic graphical model, is one of the most popular topic models due to its interpretability and effectiveness. however, the parameter estimation methods for lda and its variants, such as collapsed gibbs sampling @xcite , are model-specific and require specialized derivations. to tackle such disadvantages, neural topic models have been proposed with a flexible training process, which can be divided into two categories, variational autoencoder (vae) based and generative adversarial network (gan) based. vae-based neural topic models regard the encoded latent vector as the topic distribution of the input document, then employ the decoder to reconstruct the word distribution @xcite @xcite @xcite . to address the limitation that vae-based neural topic models cannot approximate dirichlet distribution precisely, @xcite propose an adversarial topic model, in which the topic distribution is sampled from the dirichlet prior distribution directly and transformed into the word distribution by the generator. in order to uncover topic distribution and infer document topic simultaneously, bidirectional adversarial topic model @xcite and topic modeling with cycle-consistent adversarial training @xcite have been proposed in turn. recently, a neural topic model named clntm has been proposed to apply contrastive learning to vae-based neural topic models @xcite . a data augmentation strategy is proposed to replace the salient and non-salient parts of the document representation according to word frequency information to construct positive and negative examples. although achieving promising results, it has such a disadvantage. as shown in figure 1 , due to the limitation of the unidirectional structure of vae, the encoder is optimized through the contrastive loss, instead of the decoder which generates topic-word distribution, leading to the gap between model training and evaluation. therefore, in this paper, we consider discovering topics based on cycle adversarial training and contrastive learning. as illustrated in the lower left part of figure 1 , by incorporating cycle adversarial training, topic distribution θ and word distribution x will be transformed bidirectionally, breaking the structural limitations of vae-based neural topic models. however, it is not straightforward to combine contrastive learning and cycle adversarial training. on the one hand, it is crucial to construct applicable positive samples for topic distributions. on the other hand, it is hard to improve the learning of topic-word distribution while maintaining the bidirectional mapping ability of cycle adversarial training. to overcome the above challenges, we propose a novel neural topic modeling framework based on adversarial training and contrastive learning (ntm-acl). a self-supervised contrastive loss is employed to make the generator capture similar topic information between positive pairs. the generation of topic-word distribution is improved directly, which mitigates the gap between model training and evaluation. meanwhile, a discriminative contrastive loss is designed to cooperate with supervised contrastive loss to avoid the adversarial training being undermined by the unbalance between generation and discrimination. moreover, data augmentation is applied to construct positive samples of topic distribution with the reconstruction ability of cycle generative adversarial network structure. the minimum items in the reconstructed distribution are substituted for corresponding items in the original distribution, which hasn't been explored before. we conduct extensive experiments to fully exploit the effectiveness of our proposed model. in a nutshell, the main contributions of our paper can be summarized as follows: to the best knowledge, we are the first to apply data augmentation to construct positive samples of topic distribution. our work is mainly related to two lines of research, including neural topic models and contrastive learning. ## method the overall architecture of the proposed ntm-acl is shown in figure 2 , which consists of three parts: 1) cycle adversarial training based neural topic model, which includes the generator, the encoder, and discriminators to transform topic distribution and word distribution bidirectionally; 2) topic-augmented contrastive learning, which includes the self-supervised contrastive loss and the discriminative contrastive loss to enhance the generator without affecting the adversarial training; 3) reconstruct min-term replacement, which is based on reconstruction ability of cycle generative adversarial network to create positive samples of topic distributions. ## conclusion in this paper, we have proposed ntm-acl, a novel topic modeling framework based on cycle adversarial training and contrastive learning. selfsupervised contrastive loss improves the generation of topic-word distribution that is used for the evaluation of topic modeling, while discriminative con-trastive loss keeps the balance of generation and discrimination. moreover, a novel data augmentation strategy is designed to create positive samples of topic distributions based on the reconstruction ability of cycle adversarial training. the experimental results show that the proposed method outperforms competitive baselines of different structures.