Do sequence-to-sequence VAE s learn global features of sentences? (2020)

Abstract: Autoregressive language models are powerful and relatively easy to train. However, these models are usually trained without explicit conditioning labels and do not offer easy ways to control global aspects such as sentiment or topic during generation. Bowman & al. 2016 adapted the Variational Autoencoder (VAE) for natural language with the sequence-to-sequence architecture and claimed that the latent vector was able to capture such global features in an unsupervised manner. We question this claim. We measure which words benefit most from the latent information by decomposing the reconstruction loss per position in the sentence. Using this method, we find that VAEs are prone to memorizing the first words and the sentence length, producing local features of limited usefulness. To alleviate this, we investigate alternative architectures based on bag-of-words assumptions and language model pretraining. These variants learn latent variables that are more global, i.e., more predictive of topic or sentiment labels. Moreover, using reconstructions, we observe that they decrease memorization: the first word and the sentence length are not recovered as accurately than with the baselines, consequently yielding more diverse reconstructions.

## introduction the problem of generating natural language underlies many classical nlp tasks such as translation, summarization, paraphrasing, etc. the problem is often formulated as learning a probabilistic model of sentences, then searching for probable sentences under this model. expressive language models are typically built using neural networks @xcite . whether based on lstms @xcite or transformers @xcite , language models are mostly autoregressive: the probability of a sentence is the product of the probability of each word given the previous words. @xcite built a variational autoencoder (vae) @xcite out of a sequence-tosequence architecture (seq2seq) @xcite . it generates text in a two-step process: first, a latent vector is sampled from a prior distribution; then, words are sampled from the probability distribution produced by the autoregressive decoder, conditionally on the latent vector. the goal was to encourage a useful information decomposition, where latent vectors would "explicitly model holistic properties of sentences such as style, topic, and high-level syntactic features" @xcite , while the more local correlations would be handled by the recurrent decoder. in principle, such a decomposition can be the basis for many applications. for example, using a single, unannotated corpus, it could enable paraphrasing (roy and grangier, 2019) or style transfer @xcite . for tasks requiring conditional generation such as machine translation or dialogue modeling, we could enforce a level of formality or impose a certain tone by clamping the latent vector. moreover, latent-variable models can represent multimodal distributions. thus, for these conditional tasks, the latent variable can be used as a source of stochasticity to ensure more diverse translations @xcite or answers in a dialogue @xcite . despite its conceptual appeal, bowman et al. (2016)'s vae suffers from the posterior collapse problem: early on during training, the kl term in the vae optimization objective goes to 0, such that the approximate posterior becomes the prior and no information is encoded in the latent variable. free bits are a popular workaround @xcite to ensure that the kl term is above a cer-tain level, thereby enforcing that some information about the input is encoded. but this information is not necessarily global. after all, posterior collapse can be solved trivially, without any learning, using encoders that copy parts of the inputs in the latent variable, yielding very local and useless features. in section 3, we show that encoders learn to partially memorize the first few words and the document lengths, as was first discovered by @xcite . to do so, we compare the average values of the reconstruction loss at different positions in the sentence to that of an unconditional language model. we elaborate on the negative consequences of this finding for generative models of texts. in section 4, we propose three simple variants of the model and the training procedure, in order to alleviate memorization and to yield more useful global features. in section 5, we empirically confirm that our variants produce more global features, i.e., features more predictive of global aspects of documents such as topic and sentiment. they do so while memorizing the first word and the sentence length less often, as shown in section 6. ## model and datasets firstly, we describe the vae based on the seq2seq architecture of @xcite . a document, sentence or paragraph, of l words @xmath0, . . . , x l ) is embedded in l vectors (e 1 , . . . , e l ). an lstm encoder processes these embeddings to produce hidden states: h 1 , . . . , h @xmath1, . . . , e l ) in general, the encoder produces a vector r that represents the entire document. in the original model, this vector is the hidden state of the last word @xmath2, but we introduce variants later on. this representation is transformed by linear functions l 1 and l 2 , yielding the variational parameters that are specific to each input document: these two vectors of dimension d fully determine the approximate posterior, a multivariate normal with a diagonal covariance matrix, q φ (z|x) = n (z|µ, diag(σ 2 )), where φ is the set of all encoder parameters (the parameters of the lstm, l 1 and l 2 ). then, a sample z is drawn from the approximate posterior, and the decoder, another lstm, produces a sequence of hidden states: where bos is a special token indicating the beginning of the sentence and [•; •] denotes the concatenation of vectors. finally, each hidden state at position i is transformed to produce a probability distribution of the word at position i + 1: for each document x, the lower-bound on the marginal log-likelihood (elbo) is: on the entire training set {x (1) , ., x (n ) }, the objective is: ## encoders partially memorize the first words and sentence length the elbo objective trades off the kl term against the reconstruction term. to minimize the objective, it is worth increasing the kl term only if the reconstruction term is decreased by the same amount or more. with free bits, the encoder is allowed to store a fixed amount of information for free. the objective becomes to minimize the reconstruction cost using the "free storage" as efficiently as possible. there are many solutions to this objective that are undesirable. for instance, we could program an encoder that would encode the words into the latent variable losslessly, until all the free bits are used. however, this model would not be more useful than a standard, left-to-right autoregressive models. therefore, it is necessary to check that such useless, purely local features are not learned. in order to visualize what information is stored in the latents, our method is to look at where gains are seen in the reconstruction loss. since the loss is a sum over documents and positions in these documents, these gains could be concentrated: i) on certain documents, for example, on large documents or documents containing rarer words; ii) at certain positions in the sentence, for example, in the beginning or in the middle of the sentence. we investigate the latter possibility. foot_0 ## improving existing models what architectures could avoid learning to memorize? we investigate simple variants and for a more thorough comparison with existing models, we refer to appendix d.1. our first variant uses a simple bag-of-words (bow) encoder in place of the lstm encoder. the sentence representation is r @xmath3, where the exponents denote components, and the indices denote positions in the sentence. we call it bowmax-lstm. it is similar to the max-pooling model of @xcite except that the maximum is taken over embeddings rather than lstm hidden states. as @xcite reported, the max-pooling operator is better than the average operator, both when the encoder is a lstm and bow (possibly because the maximum introduces a non-linearity). therefore, we use the maximum operator. a priori, we think that since word order is not provided to the encoder, the encoder should be unable to memorize the first words. for our second variant we use a unigram decoder (uni) in place of an lstm decoder. it produces a single output probability distribution 3 or by explicitly conditioning on the sentence length. it can be useful for unsupervised summarization @xcite , in flow-based approaches (ziegler and rush, 2019), or more broadly for the decoder to plan sentence construction. both the bow encoders and uni decoders variants might benefit from the preae pretraining technique, but we leave this for future work. lastly, the pretrained lm (prelm) variant is obtained in two training steps. first, we pretrain a lstm-lm. then, it is used as an encoder with fixed weights. we use average pooling over the hidden states to get a sentence representation, i.e., @xmath4, and learn the transformations l 1 and l 2 that compute the variational parameters. initially, we tried to use max-pooling but the training was extremely unstable. the lm objective requires the hidden state to capture both close correlations between words but also more global information to predict long-distance correlations. the hope is that this global information can be retrieved via pooling and encoded in the variational parameters. the prelm variant is nothing more than the use of a pretrained lm as a feature extractor @xcite . while @xcite and @xcite both consider the use of pretrained lms as encoders, the weights are not frozen such that it is hard to disentangle the impact of pretraining from subsequent training. in contrast, we freeze the weights so that the effect of pretraining can not be overridden. to isolate the effect of this training procedure independently of the architecture, we keep the same lstm instead of using more powerful architectures such as transformers. ## semi-supervised learning evaluation we turn to the semi-supervised learning (ssl) setting to compare the learned representations of our variants. for the purpose of controllable text generation, we assume that the global information that is desirable to capture is the topic or sentiment. there are two training phases: first, an unsupervised pretraining phase where vaes are trained; second, a supervised learning phase where classi-fiers are trained to predict ground-truth labels given the latent vectors encoded with the encoders of the vaes. this is essentially the same setup as m1 from @xcite . 5 the small and large data-regimes give us complementary information: with many labels and complex classifiers, we quantify how much of the information pertaining to the labels is encoded; with few labels and simple classifiers, how accessible the information is. for each dataset, we subsample @xmath5, containing 5, 50, 500, and 5000 examples per class. these labeled datasets are used for training and validating during the supervised learning phase. 6 each model is trained with @xmath6. the performance of the classifiers are measured by the macro f1-score on the entire test sets. to select hyperparameters on each subsample, we use repeated stratified k-fold cross-validation @xcite as detailed in the appendix e.1. we obtain the test set f1-scores f ij , where i is the subsample seed and j is the parameter initialisation seed, and report f•• , the average f1-score over i and j. we note f•j the empirical average f1-score for a given parameter initialisation j and decompose the variance into: in the context of anova with a linear model and a single factor, these quantities are the square roots of m s t and m s e (see appendix e.2). finally, we also add a data-regime where the entire labeled training set is used in the supervised learning phase. in this setting, we use more expressive one-hidden-layer mlp classifiers, with early stopping on the validation set. thus, we can check that our conclusions in the large data-regime do not depend on the model selection procedure and the choice of the classifier. 5 we could integrate the labels into the generative model as a random variable that is either observed or missing to obtain better results @xcite . still, our goal is to study the inductive bias of the seq2seq vae as an unsupervised learning method, so we do not train the encoder using the labels. for each class of model, we perform a grid search over target rates and latent vector sizes. we search for target rates λ in {2, 8}: large enough to capture label information but small enough to avoid underfitting, as explained above. the size of latent vectors d are chosen in {4, 16}. they should be small enough for extremely low-data regimes. for instance, on yelp, the smallest data regime (5 per class) uses only 8 examples to train the classifier and 2 to do cross-validation. a thorough explanation is presented in appendix c, along with the values of hyperparameters held constant. what representation should be used as inputs to the classifiers? @xcite use samples from the approximate posterior q φ (z|x) = n (z|µ, diag(σ 2 )), but in the nlp literature, most evaluations focus on µ without mention or justification. to evaluate the vae as a generative model, we claim that only noisy samples z should be used. in fact, using a model with a rate close to 0 on yelp, we can recover the label with a high f1-score of 81.5% by using µ, whereas, as expected, noisy samples z do not do better than random (50%). the information contained in µ is misleading because it is not transmitted to the decoder and not used directly during generation. therefore, we use samples z (cf. appendix e.3 for details). ## text generation evaluation how do these different variants perform during generation? we expect that the ssl classification performances would correlate with the abilities of the decoders to reconstruct documents that exhibit a similar global aspect than the encoded documents. to measure the agreement in label between the source document and its reconstruction, we adapt the evaluation procedure used by @xcite so that no human annotators or heuristics are required (see appendix d.2). first, a classifier is trained to predict the label on the source dataset. then, for each model, we encode the documents, reconstruct them, and classify these reconstructions using the classifier. the agreement is the f1-scores between the original labels and the labels given by the classifiers on the generated samples. to quantify memorization, we measure the reconstruction accuracy of the first word and the ratio of identical sentence length between sources and reconstructions. finally, to verify that our bag-ofwords assumptions do not hurt the overall fit to the data, we estimate the negative log-likelihood via the importance-weighted lower bound @xcite (500 samples) to compute an approximate perplexity per word (≈ppl). we use two decoding schemes: beam search with a beam of size 5 and greedy decoding. we fix λ = 8, @xmath7, with three seeds. for the uni decoder, we drop lstm-last-uni which underperformed by a large margin in the ssl setting, and for the other uni models, we freeze the encoder, l 1 and l 2 and train a new recurrent decoder using the reconstruction loss only. essentially, the uni decoder is an auxiliary decoder, as described by de fauw et al. (2019 ## conclusion eliminating posterior collapse is necessary to get useful vae models, but not sufficient. although recent incarnations of the seq2seq vae fix the posterior collapse, they partially memorize the first few words and the document lengths. depending on the data, these local features are sometimes not very correlated with global aspects like topic or sentiment. therefore, they are of limited use for controllable and diverse text generation. to learn to infer more global features, we ex-plored alternative architectures based on bag-ofword assumptions on the encoder or decoder side, as well as a pretraining procedure. these variants are all effective, in particular, the unigram decoder used as an auxiliary decoder @xcite . the latent variable is more predictive of global features and memorisation of the first words and sentence length is decreased. thus, these models are more suitable for diverse and controllable generation. methodologically, we introduced a simple way to examine the content of latent variables by looking at the reconstruction loss per position. we also presented a reliable way to perform semisupervised learning experiments to analyze the content of the variable, free of the problems that one can find in past work (incorrect model selection for small data-regimes, use of samples instead of variational parameters as inputs). we showed that there are particularly difficult datasets for which the first words are not very predictive of their labels, and therefore, these datasets should be systematically used in evaluations. moreover, the agreement metric is another complementary evaluation that is automatic and focused on generation. we hope that these methods will see widespread adoption for measuring progress more reliably. a promising research direction is to investigate the root cause behind memorization. a simple reason for the memorization of the first few words could be that, in the beginning of training, the reconstruction loss is higher on these words (see lstm-lm in figures 1 , 2 , 3 , 4 ). these early errors should therefore account for a proportionally large part of the gradients and pressure the encoder to store information about the first words. if that is correct, the left-to-right factorization of the decoder could be at fault, which would explain the successes of the unigram decoders. more powerful decoders with alternative factorizations could avoid this issue, for example, non-autoregressive transformers @xcite or transformers with flexible word orders @xcite . vaes operate on uncorrupted inputs and learn a corruption process in the latent space. in contrast, models in the bert family @xcite are given corrupted inputs and are penalized only on these corrupted inputs, thereby avoid memorization altogether. therefore, another research avenue would be to blend the two frameworks @xcite . a. beygelzimer, f. d'alché buc, e. fox, and r. garnett, editors, advances in neural information processing systems 32, pages 8024-8035. curran associates, inc. tom pelsmaeker and wilker aziz. 2019. effective estimation of deep generative language models. arxiv preprint arxiv:1904.08194. matthew e peters, mark neumann, mohit iyyer, matt gardner, christopher clark, kenton lee, and luke zettlemoyer. 2018. deep contextualized word representations. arxiv preprint arxiv:1802.05365. victor prokhorov, ehsan shareghi, yingzhen li, mohammad taher pilehvar, and nigel collier. 2019. on the importance of the kullback-leibler divergence term in variational autoencoders for text generation. in proceedings of the 3rd workshop on neural generation and translation, pages 118-127, hong kong. association for computational linguistics. alec radford, jeffrey wu, rewon child, david luan, dario amodei, and ilya sutskever. 2019. language models are unsupervised multitask learners. technical report. ali razavi, aaron van den oord, ben poole, and oriol vinyals. 2019. preventing posterior collapse with delta-vaes. in international conference on learning representations. danilo jimenez rezende, shakir mohamed, and daan wierstra. 2014. stochastic backpropagation and approximate inference in deep generative models. in international conference on machine learning, pages 1278-1286. aurko roy and david grangier. 2019. unsupervised paraphrasing without translation. arxiv preprint arxiv:1905.12752. raphael schumann. 2018. unsupervised abstractive sentence summarization using length controlled variational autoencoder. arxiv preprint arxiv:1809.05233. iulian vlad serban, alessandro sordoni, ryan lowe, laurent charlin, joelle pineau, aaron courville, and yoshua bengio. 2017. a hierarchical latent variable encoder-decoder model for generating dialogues. in thirty-first aaai conference on artificial intelligence. xing shi, kevin knight, and deniz yuret. 2016. why neural translations are the right length. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 2278-2282, austin, texas. association for computational linguistics. paul smolensky. 1990. tensor product variable binding and the representation of symbolic structures in connectionist systems. artificial intelligence, 46(1-2):159-216. martin sundermeyer, ralf schlüter, and hermann ney. 2012. lstm neural networks for language modeling. in thirteenth annual conference of the international speech communication association. ilya sutskever, oriol vinyals, and quoc v le. 2014. sequence to sequence learning with neural networks. in z. ghahramani, m. welling, c. cortes, n. d. lawrence, and k. q. weinberger, editors, advances in neural information processing systems 27, pages 3104-3112. curran associates, inc. mirac suzgun, yonatan belinkov, stuart shieber, and sebastian gehrmann. 2019. lstm networks can perform dynamic counting. in proceedings of the workshop on deep learning and formal languages: building bridges, pages 44-54, florence. association for computational linguistics. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. in i. guyon, u. v. luxburg, s. bengio, h. wallach, r. fergus, s. vishwanathan, and r. garnett, editors, advances in neural information processing systems 30, pages 5998-6008. curran associates, inc. paul j werbos. 1990. backpropagation through time: what it does and how to do it. proceedings of the ieee, 78(10):1550-1560. peng xu, yanshuai cao, and jackie chi kit cheung. 2019. unsupervised controllable text generation with global variation discovery and disentanglement. arxiv preprint arxiv:1905.11975. zichao yang, zhiting hu, ruslan salakhutdinov, and taylor berg-kirkpatrick. 2017. improved variational autoencoders for text modeling using dilated convolutions. in proceedings of the 34th international conference on machine learning-volume 70, pages 3881-3890. jmlr. org. xiang zhang, junbo zhao, and yann lecun. 2015. character-level convolutional networks for text classification. in c. cortes, n. d. lawrence, d. d. lee, m. sugiyama, and r. garnett, editors, advances in neural information processing systems 28, pages 649-657. curran associates, inc. tiancheng zhao, ran zhao, and maxine eskenazi. 2017. learning discourse-level diversity for neural dialog models using conditional variational autoencoders. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 654-664, vancouver, canada. association for computational linguistics. zachary ziegler and alexander rush. 2019. latent normalizing flows for discrete sequences. in international conference on machine learning, pages 7673-7682. a on the use of kl annealing, the choice of the free bits flavor and resetting the decoder @xcite evaluated their models in the ssl setting (section 3.3 of their paper). however, their experimental setting is not very rigorous. in the case of the 100 labeled examples, hyperparameter selection is done on a very large validation set of 10000 examples. however, the validation set here should be seen as nothing more than a split of the training data dedicated to optimising hyperparameters. in the words of cawley and talbot (2010), "model selection should be viewed as an integral part of the model fitting procedure". besides methodological issues, we run our own hyperparameter search on the yelp dataset to properly disentangle the effects of kl annealing, the free bits method and verify the importance of resetting the decoder. we use the semi-supervised learning setting presented in section 5 to evaluate the learned encoders.