Challenging the Semi-Supervised VAE Framework for Text Classification (2021)

Abstract: Semi-Supervised Variational Autoencoders (SSVAEs) are widely used models for data efficient learning. In this paper, we question the adequacy of the standard design of sequence SSVAEs for the task of text classification as we exhibit two sources of overcomplexity for which we provide simplifications. These simplifications to SSVAEs preserve their theoretical soundness while providing a number of practical advantages in the semi-supervised setup where the result of training is a text classifier. These simplifications are the removal of (i) the Kullback-Liebler divergence from its objective and (ii) the fully unobserved latent variable from its probabilistic model. These changes relieve users from choosing a prior for their latent variables, make the model smaller and faster, and allow for a better flow of information into the latent variables. We compare the simplified versions to standard SSVAEs on 4 text classification tasks. On top of the above-mentioned simplification, experiments show a speed-up of 26%, while keeping equivalent classification scores. The code to reproduce our experiments is public.

## introduction obtaining labeled data to train nlp systems is a process that has often proven to be costly and time-consuming, and this is still largely the case @xcite . consequently, semi-supervised approaches are appealing to improve performance while alleviating dependence on annotations. to that end, variational autoencoders (vaes) @xcite have been adapted to semi-supervised learning @xcite , and subsequently applied to several nlp tasks @xcite @xcite . a notable difference between the generative model case from where vaes originate, and the 1 https://github.com/ghazi-f/challenging-ssvaes semi-supervised case is that only the decoder (generator) of the vae is kept after training in the first case, while in the second, it is the encoder (classifier) that we keep. this difference, as well as the autoregressive nature of text generators has not sufficiently been taken into account in the adaptation of vaes to semi-supervised text classification. in this work, we show that some components can be ablated from the long used semi-supervised vaes (ssvaes) when only aiming for text classification. these ablations simplify ssvaes and offer several practical advantages while preserving their performance and theoretical soundness. the usage of unlabeled data through ssvaes is often described as a regularization on representations @xcite @xcite . more specifically, ssvaes add to the supervised learning signal, a conditional generation learning signal that is used to train on unlabeled samples. from this observation, we study two changes to the standard ssvae framework. the first simplification we study is the removal of a term from the objective of ssvaes: the kullback-leibler term. this encourages the flow of information into latent variables, frees the users from choosing priors for their latent variables, and is harmless to the theoretical soundness of the semisupervised framework. the second simplification we study is made to account for the autoregressive nature of text generators. in the general case, input samples in ssvaes are described with two latent variables: a partially-observed latent variable, which is also used to infer the label for the supervised learning task, and an unobserved latent variable, which describes the rest of the variability in the data. however, autoregressive text generators are powerful enough to converge without the need for latent variables. therefore, removing the unobserved latent variable is the second change we study in ssvaes. the above modifications can be found in some rare works throughout the literature, e.g. @xcite . we, however, aim to provide justification for these changes beyond the empirical gains that they exhibit for some tasks. our experiments on four text classification datasets show no harm to the empirical classification performance of ssvae in applying the simplifications above. additionally, we show that removing the unobserved latent variable leads to a significant speed-up. to summarize our contribution, we justify two simplifications to the standard ssvae framework, explain the practical advantage of applying these modifications, and provide empirical results showing that they speed up the training process while causing no harm to the classification performance. ## experiments in this section, we display comparisons between instances of standard ssvaes and the same ssvaes after applying the changes we propose. ## related works after the pioneering work of @xcite , ssvaes were extended to tasks such as morphological inflections @xcite , controllable speech synthesis @xcite , parsing @xcite , sequential labeling @xcite among many others. vae internals have also been tweaked in various manners to improve the learning performance. for instance, @xcite introduce a low resource pretraining scheme to improve transfer with vaes, while @xcite propose to use the deterministic ancestor of a latent variable to perform classification, and constrain it with an adversarial term to have it abide by the values of the random latent variable. while our work is a focused contribution dedicated to the theoretical soundness and the practical advantages of two simplifications to the ssvae framework for text classifications, it could be extendend to other tasks involving text generation as the unsupervised vae objective. for instance, the work of @xcite shows that semisupervised dependency parsing scores higher with both the changes we study. ## conclusion starting from the observation that ssvaes can be viewed as the combination of a supervised learning signal with an unsupervised conditional generation learning signal, we show that this framework needs neither to include a kl-divergence nor an unobserved latent variable (z) when dealing with text classification. we subsequently perform experimental comparisons between standard ssvaes and simplified ssvaes that indicate that they are globally equivalent in performance. our changes provide a number of practical advantages. first, removing the kl-divergence frees practitioners from choosing priors for the variables they use, and allows information to flow freely into these variables. second, removing the latent variable z from the computational graph speeds up computation and shrinks the size of the network. despite their popularity, vaes are often tedious to train for nlp tasks. in that regard, our simplifications should facilite their usage in future works.