Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling (2023)

Abstract: Dropout is a widely used regularization trick to resolve the overfitting issue in large feedforward neural networks trained on a small dataset, which performs poorly on the held-out test subset. Although the effectiveness of this regularization trick has been extensively studied for convolutional neural networks, there is a lack of analysis of it for unsupervised models and in particular, VAE-based neural topic models. In this paper, we have analyzed the consequences of dropout in the encoder as well as in the decoder of the VAE architecture in three widely used neural topic models, namely, contextualized topic model (CTM), ProdLDA, and embedded topic model (ETM) using four publicly available datasets. We characterize the dropout effect on these models in terms of the quality and predictive performance of the generated topics.

## introduction dropout @xcite is used while training neural networks, by stochastically dropping out the activation of neurons to prevent complex co-adaptations of feature vectors @xcite . the working of dropout is attributed to the implicit averaging over an ensemble of neural networks @xcite . it has been shown to be effective on supervised learning tasks to prevent overfitting @xcite . as the volume of digital documents significantly increases with time, organizing them manually is becoming quite an inconvenient task. because of the ability of topic models to learn a thematic structure from a set of documents in an unsupervised manner and label the documents with their corresponding dominant topics, the significance of topic models is enormous in this area @xcite . but in the traditional topic models, not only the computation cost of the approximate posterior is very high but also for a small change in the modeling assumption, re-derivation of the inference method is needed. with greater flexibility and scalability than traditional topic models, a class of neural topic models (ntms) aim to leverage the potential of neural networks using the aevb (kingma and welling, 2014) based inference technique. following @xcite , we refer to this class of models as vae-ntms where the training objective is to maximize the log-likelihood of the reconstruction of the input document while minimizing the kl-divergence of the learned posterior distribution of the latent space from a known prior distribution. an earlier study by @xcite of the dropout effect on two traditional topic models lda @xcite and btm @xcite shows that the correct choice of the dropout rate not only decreases the learning time of the models but also significantly improves the predictive performance and generalization for short texts. however, the study does not consider neural topic models. in this work, we propose the use of dropout on vae-ntms as a hyperparameter in order to achieve much better performance in terms of topic coherence, topic diversity, and topic quality. we test this proposition on a range of standard vae-ntm architectures. to the best of our knowledge, there has been no other study focusing specifically on the use of dropout in neural topic models. we have made our analysis publicly available 1 . in summary, our contributions are as follows: ## task formulation given a corpus {d 1 , d 2 , . . . , d n } of n documents with vocabulary {w 1 , w 2 , ..., w v } of v words, topic models describe a document d i as a distribution over k topics {β 1 , β 2 , ..., β k }, where an individual topic β k is a distribution over vwords. ## empirical study we perform all experiments in octis @xcite , which is an integrated framework for topic modeling. ## theoretical understanding of results our experiments show that by tuning the dropout carefully, we can achieve a significant improvement in the performance of vae-ntms. therefore, we argue that the dropout rate should be treated as an important hyperparameter and carefully selected based on the choice of the model as well as the dataset, especially in the case of vae-ntms. more precisely, in most cases, low dropout rates in the encoder and the decoder lead to higher performance than that achieved for higher dropout rates. standard dropout and other types of dropout have been extensively used in supervised learning techniques @xcite @xcite @xcite . the main prerogative of using dropout in the supervised scenario is to introduce noise while training so that the model can recognize the outliers in the testing phase. the drop in performance with high dropout that we see in our experiments is perhaps due to the fact that we are trying to learn a generative model of the data. dropout makes the model robust against perturbations in the input data and thereby also prevents it from learning the characteristics of the input distribution accurately. this is probably why we see a drop in topic coherence and quality. in the case of document classification, if the topic model is trained with a high dropout, the document-topic vectors are of poor quality and the classifier gets trained on these vectors; this results in poor accuracy on the test documents. this setting is different from the usual supervised learning of neural classifiers where dropout is introduced directly in the classifier to prevent overfitting. we intend to analyze these aspects in more depth in the future. ## conclusion we present a detailed study of the effect of the dropout rate on vae-ntms. we find that the model performance generally reduces with the increase in dropout rate in the encoder as well as the decoder.