Reinforcement Learning for Topic Models (2023)

Abstract: We apply reinforcement learning techniques to topic modeling by replacing the variational autoencoder in ProdLDA with a continuous action space reinforcement learning policy. We train the system with a policy gradient algorithm REINFORCE. Additionally, we introduced several modifications: modernize the neural network architecture, weight the ELBO loss, use contextual embeddings, and monitor the learning process via computing topic diversity and coherence for each training step. Experiments areperformed on 11 data sets. Our unsupervised model outperforms all other unsupervised models and performs on par with or better than most models using supervised labeling. Our model is outperformed on certain data sets by a model using supervised labeling and contrastive learning. We have also conducted an ablation study to provide empirical evidence of performance improvements from changes we made to ProdLDA and found that the reinforcement learning formulation boosts performance. We open-source our code implementation.

## introduction the internet contains large collections of unlabeled textual data. topic modeling is a method to extract information from this text by grouping documents into topics and linking these topics with words describing them. classical techniques for topic modeling, the most popular being latent dirichlet approximation (lda) @xcite , have recently begun to be overtaken by neural topic models (ntm) @xcite . prodlda @xcite is a ntm using a product of experts in place of the mixture model used in classical lda. prodlda uses a variational autoencoder (vae) @xcite to learn distributions over topics and words. prodlda improved on nvdm (miao 1 https://github.com/jeremy-costello/rl-for-topic-models et al., 2016) by explicitly approximating the dirichlet prior from lda with a gaussian distribution and using the adam optimizer (kingma and ba, 2014) with a higher momentum and learning rate. perceiving reinforcement learning (rl) as probabilistic inference has brought practices of such an inference into the rl field @xcite . new algorithms using these techniques include mpo @xcite and virel @xcite . mpo optimizes the evidence lower bound (elbo), which is the same optimization objective used in vaes. inspired by the adoption of probabilistic inference techniques in rl, we look to apply rl techniques to probabilistic inference in the realm of topic models. we use reinforce, the simplest policy gradient (pg) algorithm, to train a model which parameterizes a continuous action space, corresponding to the distribution of topics for each document in the topic model. we keep the product of experts from prodlda to compute the distribution of words for each document in the topic model. we additionally improve our topic model by using sentence-bert (sbert) embeddings @xcite rather than bag-ofword (bow) embeddings, modernizing the neural network (nn) architecture, adding a weighting term to the elbo, and tracking topic diversity and coherence metrics throughout training. the model architecture is shown in figure 1 . our method outperforms most other topic models. it is beaten only on some data sets by advanced methods using document labels for supervised learning, while our procedure is fully unsupervised. our approach is a modification of the prodlda model @xcite . the novelty of our approach is as follows: zhao et al. (2021) provide a survey of ntms. variations of vaes are presented which use different distributions, correlated and structured topics, pretrained language models, incorporate meta-data, or model on short texts rather than documents. methods other than vaes are also used for ntming, including autoregressive models, generative adversarial networks, and graph nns. @xcite compare prodlda and nvdm, along with six other ntms and three classical topic models, in terms of held-out document and word perplexity, downstream classification, and coherence. scholar @xcite , an extension of prodlda taking document metadata and labels into account where possible, performed best in terms of coherence. nvdm and nvctm @xcite , an extension of nvdm which additionally models the correlation between documents, performed best in terms of perplexity and downstream classification. the other ntms were gsm @xcite , nvlda @xcite , nsmdm @xcite , and nsmtm @xcite . the classical topic models were non-negative matrix factorization (nmf) @xcite , online lda @xcite , and gibbs sampling lda @xcite . bertopic @xcite and top2vec @xcite use dimensionality reduction and clustering to group document embeddings from pre-trained language models into meaningful clusters. contextualized topic models (ctm) @xcite augments the bow embeddings used in prodlda with sbert @xcite embeddings, resulting in an improved topic model. @xcite develop the embedded topic model (etm) by using word embeddings to augment a variational inference algorithm for topic modeling. their method outperforms other topic models, especially on corpora with large vocab-ularies containing common and very rare words. @xcite augment scholar @xcite with contrastive learning @xcite and outperform all topic models compared against. @xcite use rl to filter words from documents, with reward as a combination of the resulting topic model's coherence and diversity, or how few words overlap between topics. @xcite , a pg rl algorithm, to augment prodlda. their model slightly outperforms prodlda in terms of topic coherence. @xcite use rl to augment a non-negative matrix factorization topic model. ## background we briefly outline topic models, rl process, kl divergence, and contextual embeddings. ## discussion for the initial experiments on the 20 newsgroups data set, the average loss (figure 2 ) reaches a near plateau around the 200th epoch. past this epoch, coherence (figure 3 ) continues to increase slowly, and topic diversity (figure 4 ) increases substantially until around the 400th epoch, past which it also continues to increase slowly. it shows that training beyond a plateau in loss can still improve ntm performance. compared to @xcite , the rl model performs on par with or better than other models across all four data sets, while the performance of other models varies greatly between data sets. on the snippets, 20 newsgroups, and w2efigure 6: dropout sweep for 20 newsgroups content data sets, the rl model with lower values of λ usually performs better as the number of topics increases. however, it reverses on the w2e-title data set where λ = 10 outperforms λ = 1 on the two highest number of topics. the rl model outperforms the labeled etm model from dieng et al. (2020) in topic diversity, coherence, and quality. furthermore, this comparison had no pruning of stop words, showing the rl model can deal with vocabularies containing many common words. newsgroups data set. if we consider models from nguyen and luu (2021), our rl model performs similarly on 50 topics but worse on 200 topics. the rl model's performance on larger topic sizes and vocabularies could be improved by adding supervised labels, applying contrastive learning, scaling up inference layer sizes, or performing a hyperparameter sweep with 200 topics. topic diversity and coherence values should be provided when reporting topic model performance. in figure 6 , the highest topic quality is achieved when there is no policy dropout. topic diversity can be sacrificed for some gain in coherence. applications of topic models may want to maximize topic diversity, coherence, or both. the description of topic model performance should reflect this. in the ablation study, removing the rl policy formulation causes the model to perform worse than the original one. it confirms rl policy augments the improvements from other changes to the model. performance suffers the most when the softmax distribution is re-added to the topic distribution during training. to recapture the softmax distribution of topics, it can be applied to the topic distribution during inference. adding policy dropout significantly reduces topic diversity and leads to a slight coherence reduction. performance improves with sbert embeddings, and the model can still recon-struct the bow within the elbo without direct access. increasing λ to 5 improves performance, but as seen from other experiments, this is only sometimes the case. ## conclusion inspired by the introduction of probabilistic inference techniques to rl, we take the approach to develop a ntm augmented with rl. our model builds on the prodlda model, which uses a product of experts instead of the mixture model used in classical lda. we improve prodlda by adding sbert embeddings, an rl policy formulation, a weighted elbo loss, and the improved nn architecture. in addition, we track topic diversity and coherence during a training process rather than only evaluating these metrics for the final model. our fully unsupervised rl model outperforms most other topic models. it is only topped by contrastive scholar -a method using supervised labels during training -in a few select cases.