A Joint Learning Approach for Semi-supervised Neural Topic Modeling (2022)

Abstract: Topic models are some of the most popular ways to represent textual data in an interpret-able manner. Recently, advances in deep generative models, specifically auto-encoding variational Bayes (AEVB), have led to the introduction of unsupervised neural topic models, which leverage deep generative models as opposed to traditional statistics-based topic models. We extend upon these neural topic models by introducing the Label-Indexed Neural Topic Model (LI-NTM), which is, to the extent of our knowledge, the first effective upstream semi-supervised neural topic model. We find that LI-NTM outperforms existing neural topic models in document reconstruction benchmarks, with the most notable results in low labeled data regimes and for data-sets with informative labels; furthermore, our jointly learned classifier outperforms baseline classifiers in ablation studies.

## introduction topic models are one of the most widely used and studied text modeling techniques, both because of their intuitive generative process and interpretable results @xcite . though topic models are mostly used on textual data @xcite , use cases have since expanded to areas such as genomics modeling @xcite and molecular modeling @xcite . recently, neural topic models, which leverage deep generative models have been used successfully for learning these probabilistic models. a lot of this success is due to the development of variational autoencoders @xcite which allow for inference of intractable distributions over latent variables through a back-propagation over an inference network. furthermore, recent research shows promising results for neural topic models compared to traditional * equal contribution topic models due to the added expressivity from neural representations; specifically, we see significant improvements in low data regimes @xcite . joint learning of topics and other tasks have been researched in the past, specifically through supervised topic models @xcite @xcite . these works are centered around the idea of a prediction task using a topic model as a dimensionality reduction tool. fundamentally, they follow a downstream task setting (figure 1 ), where the label is assumed to be generated from the latent variable (topics). on the other hand, an upstream setting would be when the input (document) is generated from a combination of the latent variable (topics) and label, which has the benefit of better directly modeling how the label affects the document, resulting in topic with additional information being injected from the label information. upstream variants of supervised topic models are much less common, with, to the extent of our knowledge, no neural architectures to this date. @xcite . our model, the label-indexed neural topic model (li-ntm) stands uniquely with respect to all existing topic models. we combine the benefits of an upstream generative processes (figure 1 ), label-indexed topics, and a topic model capable of semi-supervised learning and neural topic modeling to jointly learn a topic model and label classifier. our main contributions are: ## experimental setup we perform an empirical evaluation of li-ntm with two corpora: a synthetic dataset and ag news. ## synthetic data experimental results we used our synthetic dataset to examine the performance of li-ntm relative to etm in a setting where the label strongly partitions our dataset into subsets that have distinct topics to investigate the effect and robustness of label indexing. li-ntm was trained on the fully labeled version of the both the ideal and worse case label synthetic dataset and etm was trained on the same dataset with the label excluded, as etm is a unsupervised method. we varied the number of topics in both lintm and etm to explore realistic settings @xmath0, 8 and the extreme setting @xmath1. ## ag news experimental results we used the ag news dataset to evaluate the performance of li-ntm in the semi-supervised setting. specifically, we aimed to analyze the extent to which unlabeled data can improve the performance of both the classifier and topic model in the li-ntm architecture. ideally, in the unlabeled case, the distant supervision provided to the classifier from the reconstruction loss would align with the task of predicting correct labels. we ran four experiments on etm and li-ntm in which the amount of unlabeled data was gradually increased, while the amount of labeled data was kept fixed. in each of the experiments, 5% of the dataset was considered labeled, while 5%, 15%, 55%, and 95% of the whole dataset was considered unlabeled in each of the four experiments respectively. ## conclusion in this paper, we introduced the li-ntm, which, to the extent of our knowledge, is the first upstream neural topic model with applications to a semisupervised data setting. our results show that when applied to both a synthetic dataset and ag news, li-ntm outperforms etm with respect to perplexity. furthermore, we found that the classifier in li-ntm was able to outperform a baseline that doesn't leverage any unlabeled data. even more promising is the fact that the classifier in li-ntm continued to experience gains in accuracy when increasing the proportion of unlabeled data. while we aim to iterate upon our results, our current findings indicate that li-ntm is comparable with current state-of-the-art models while being applicable in a wider range of real-world settings. in future work, we hope to further experiment with the idea of label-indexing. while in li-ntm every topic is label-specific, real datasets have some common words and topics that are labelagnostic. future work could augment the existing li-ntm framework with additional label-agnostic global topics which prevent identical topics from being learned across multiple labels. we are also interested in extending our semi-supervised, upstream paradigm to a semi-parametric setting in which the number of topics we learn is not a predefined hyperparameter but rather something that is learned. figure a1 : the probabilities of the top words from 5 selected topics from lint-m. note that lint-m has the nice property that topics are naturally sorted by label unlike etm. the first topic, with words like "series", "yankees", "red", "sox", corresponds to baseball. note that perplexity will still be high even if this topic is correctly given a high proportion in a baseball-themed news article since there are many potential baseball teams and baseball terminology that the article could be referencing. the second topic corresponds to search engines, and the third corresponds to the israeli-palestinian conflict.