Discrete Latent Variable Representations for Low-Resource Text Classification (2020)

Abstract: While much work on deep latent variable models of text uses continuous latent variables, discrete latent variables are interesting because they are more interpretable and typically more space efficient. We consider several approaches to learning discrete latent variable models for text in the case where exact marginalization over these variables is intractable. We compare the performance of the learned representations as features for low-resource document and sentence classification. Our best models outperform the previous best reported results with continuous representations in these low-resource settings, while learning significantly more compressed representations. Interestingly, we find that an amortized variant of Hard EM performs particularly well in the lowest-resource regimes.

## introduction deep generative models with latent variables have become a major focus of nlp research over the past several years. these models have been used both for generating text @xcite and as a way of learning latent representations of text for downstream tasks @xcite . most of this work has modeled the latent variables as being continuous, that is, as vectors in r d , in part due to the simplicity of performing inference over (certain) continuous latents using variational autoencoders and the reparameterization trick @xcite . at the same time, deep generative models with discrete latent variables are attractive because the latents are arguably more interpretable, and because they lead to significantly more compressed representations: a representation consisting of m floating point values conventionally requires m × 32 bits, whereas m integers in {1, . . . , k} requires only m × log 2 k bits. unfortunately, discrete latent variable models have a reputation for being more difficult to learn. we conduct a thorough comparison of several popular methods for learning such models, all within the framework of maximizing the evidence lower bound (elbo) on the training data. in particular, we compare learning such models either with a vector quantized-vae @xcite , a more conventional vae with discrete latent variables @xcite , or with an amortized version of "hard" or "viterbi" expectation maximization @xcite , which to our knowledge has not been explored to date. we consider both models where the latents are local (i.e., per token) and where they are global (i.e., per sentence); we assess the quality of these learned discrete representations as features for a low-resource text classifier, as suggested by @xcite , and in a nearest neighborbased retrieval task. our classification experiments distinguish between (1) the setting where the classifier must consume only the discrete representation associated with each sentence (i.e., the discrete assignment that maximizes the approximate posterior), and (2) the setting where the classifier may consume the embeddings of this discrete representation learned by the vae encoder. note that the former setting is more flexible, since we need only store a sentence's discrete representation, and are therefore free to use task-specific (and possibly much smaller) architectures for classification. in case (1), we are able to effectively match the performance of @xcite and other baselines; in case (2), we outperform them. our experiments also suggest that hard em performs particularly well in case (1) when there is little supervised data, and that vq-vae struggles in this setting. ## related work our work builds on recent advances in discrete representation learning and its applications. in particular, we are inspired by recent success with vq-vaes outside nlp @xcite . these works show that we can generate realistic speech and image samples from discrete encodings, which better align with symbolic representations that humans seem to work with (e.g., we naturally encode continuous speech signals into discrete words). despite its success in speech and vision, vq-vae has not been considered as much in nlp. one exception is the translation model of @xcite that encodes a source sequence into discrete codes using vector quantization. but their work focuses on making inference faster, by decoding the target sequence from the discrete codes non-autoregressively. to our knowledge, we are the first that explores general text representations induced by vq-vaes for semi-supervised and transfer learning in nlp. in addition to exploring the viability of vq-vaes for text representation learning, an important part of this paper is a systematic comparison between different discretization techniques. gumbel-softmax @xcite is a popular choice that has been considered for supervised text classification @xcite and dialog generation @xcite . in the binary latent variable setting, straight-through estimators are often used @xcite ). another choice is "continuous decoding" which takes a convex combination of latent values to make the loss differentiable @xcite ). yet a less considered choice is hard em @xcite @xcite . a main contribution of this work is a thorough empirical comparison between such different choices in a controlled setting. to demonstrate the usefulness of our models, we focus on improving low-resource classification performance by pretraining on unlabeled text. previous best results are obtained with continuous latentvariable vaes, e.g., vampire @xcite . we show that our discrete representations outperform these previous results while being significantly more lightweight. ## background we consider generative models of a sequence @xmath0:t of t word tokens. we assume our latents to be a sequence @xmath1:l of l discrete latent vectors, each taking a value in {1, . . . , k} m ; that is, @xmath2, . . . , k} m ×l . as is common in vae-style models of text, we model the text autoregressively, and allow arbitrary interdependence between the text and the latents. that is, we have p(x, z; θ) = p(z) × t @xmath3, z; θ), where θ are the generative model's parameters. we further assume p(z) to be a fully factorized, uniform prior: p(z) = 1 k m l . maximizing the marginal likelihood of such a model will be intractable for moderate values of k, m , and l. so we consider learning approaches that maximize the elbo @xcite in an amortized way @xcite : ## models and architectures recall that the latent sequence is @xmath4:l , where z @xmath5, . . . , k} m . we consider two generative models p(x | z; θ), one where @xmath6. each latent in the former model corresponds to a word, and so we refer to this as a "local" model, whereas in the second model we view the latents as being "global", since there is one latent vector for the whole sentence. we use the following architectures for our encoders and decoder, as illustrated in figure 1 . ## conclusion we have presented experiments comparing the discrete representations learned by a categorical vae, a vq-vae, and hard em in terms of their ability to improve a low-resource text classification system, and to allow for nearest neighbor-based document retrieval. our best classification models are able to outperform previous work, and this remains so even when we reembed discrete latents from scratch in the learned classifier. we find that amortized hard em is particularly effective in lowresource regimes when reembedding from scratch, and that vq-vae struggles in these settings.