Contrastive Deterministic Autoencoders For Language Modeling (2023)

Abstract: Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. Training VAEs, especially for text, often runs into the issue of posterior collapse, resulting in loss of representation quality. Deterministic autoencoders avoid this issue, and have been explored particularly well for images. It is however unclear how to best modify a deterministic model designed for images into a successful one for text. We show that with suitable adaptations, we can significantly improve on batch-normed VAEs (BN-VAEs), a strong benchmark for language modeling with VAEs, by replacing them with analogous deterministic models. We employ techniques from contrastive learning to control the entropy of the aggregate posterior of these models to make it Gaussian. The resulting models skip reparametrization steps in VAE modeling and avoid posterior collapse, while outperforming a broad range of VAE models on text generation and downstream tasks from representations. These improvements are shown to be consistent across both LSTM and Transformer-based VAE architectures. Appropriate comparisons to BERT/GPT-2 based results are also included. We also qualitatively examine the latent space through interpolation to supplement the quantitative aspects of the model.

## introduction the variational autoencoder (kingma and welling, 2013) -henceforth referred to simply as the vae -is a classical neural model that utilizes a paired encoder-decoder structure. for every data instance x i , the encoder network in the vae is responsible for creating a compressed code distribution p (z i |x i ) parametrically. the decoder network, in turn, uses this p (z i |x i ) to form an approximation of the real input, xi through an intermediate sampling step. by minimizing a reconstruction loss between x i and xi , along with a kl-divergence between p (z i |x i ) and the isotropic gaussian distribution, vaes can perform both generative and denoising (reconstructive) tasks. minimization of the kl loss allows the vae to create an isotropic gaussian distribution to sample and decode from. however, note that the isotropic gaussian is not required and in nlp researchers have considered training the latent distribution @xcite and learning structured, discrete representations @xcite . one of the most pressing problems in practical vae training arises when the encoder's distribution collapses to the standard gaussian for every instance, that is, p (z i |x i ) becomes ≈ n (0, i) ∀i. this problem is commonly termed as posterior collapse @xcite @xcite and lies at the heart of modern issues with vaes. many fixes have been proposed towards this problem, ranging from setting a lower bound on the kl term (δ-vae) @xcite , aggressively training the encoder and analyzing the 'lag' between the encoder and the decoder (agg-vae) @xcite , force meaningful usage of the code z i through skip connections (skip-vae) @xcite . these issues worsen when the vae employs an autoregressive structure, such as for text or videos @xcite @xcite . thus, mitigating posterior collapse in vae architectures is likely to have outsized benefits in nlp. independent of the vae model, there exists the idea of aligning the aggregate posterior. the aggregate posterior consists of the latent space distribution, i.e. the distribution over z formed by evaluating p (x i )p (z i |x i ) over all x i , where the distribution over x i is usually computed by an empirical average over the training set. in these methods, the individual distributions, p (z i |x i ) may even be dirac distributions, i.e. the mapping between z i and x i is purely deterministic. we can still find, in aggregate, a distribution over z that is close to an isotropic gaussian. methods in this vein utilize wasserstein-based optimal transport @xcite , maximum mean discrepancy @xcite , etc. for this purpose. these methods cannot truly be termed vaes, as they are often deterministic, but they work similarly. due to their deterministic nature and differing loss functions, posterior collapse does not usually occur. in vaes, the quantity to be maximized is the log likelihood log p (x i ). this proves intractable, and an equivalent optimization is done via the elbo (evidence lower bound) objective. while vaes can be evaluated by log likelihood, another metric is to evaluate their reconstruction error, as well as the quality of samples generated when, in place of p (z), an isotropic gaussian n (0, i) is substituted and the gaussian samples fed through the decoder network. this notion of sample-based evaluation can also be done for deterministic autoencoders, which do not allow an elbo evaluation. this allows us to compare deterministic autoencoders to variational options fairly to determine superiority. ## information theory of constant variance batch normed autoencoders let the generation process of the latent code be : consider an intermediate case between the vae (where each σ j can be distinct) and the deterministic autoencoder (each σ @xmath0. set every σ @xmath1, @xmath2. then, indexwise and vectorwise, where h is the entropy function. now, z|µ = (µ + z)|µ. therefore, the required entropy is of z is independent of µ. this differs from general vaes, where the variance is instance dependent and σ i , µ i are related, relating z, µ. note that here we refer to the instance-index i and not the dimension index j as we discuss instance-level dependence. this yields the final mutual information expression as : now, h(z) has a fixed value of e[z] and also a fixed variance, since it is the direct sum of two random variables z and µ, both of which, by hypothesis, have fixed means and variances. under this condition of fixed mean and variance, it is known that the entropy h is maximized iff z is distributed as a gaussian @xcite . therefore, a more informative constant variance batch normed vae induces a more gaussian representation on z. since µ = z -z, and z approaches a gaussian, while z is itself one, the desired latent mean µ also approaches a gaussian as mutual information rises. note that our analysis holds for any @xmath3. this means even very low values of cslowly annealed to 0, approximating the deterministic autoencoder case -will work as long as the mutual information required is high. to create an aggregate posterior which is the isotropic gaussian, we assume @xmath4, @xmath5, i.e. ## results in terms of text modeling, results appear in tables 1 and 2. in general, our method outperforms the competitors and if not is close to the top performer. it should be noted that realistically, the lstm critic's performance relative to gpt-2 is due to the fact the lstm is trained on the relevant corpus while gpt-2 is tested zero-shot. even though gpt-2 is a stronger model, the lstm has more domain knowledge, causing their perplexities to be close. this implies that gpt-2 evaluates the samples based on general knowledge of the english language and plausibility as english sentences (as it is tested zero-shot) while the lstm evaluates it with emphasis on the domain knowledge (which is the sole train data). having both perplexities thus evaluates differently, and we perform well on both. to evaluate the quality of the latent space for downstream tasks, we extract the latent representations in a shortened yelp dataset following @xcite , along with the labels for a small fraction of the dataset. these labels reflect the nature of the shortened review (positive or negative). we ## qualitative and quantitative analysis of interpolations autoencoders allow smooth linear interpolation between two sentences in the latent space in a manner that should allow both syntactic (sentence structure) and semantic meaning. this is captured using the subset of the short yelp review dataset, which consists of small single sentence reviews of establishments with either 0 or 1 for a negative or positive review, and is used in the classification task in results demonstrating these qualitative characteristics are summarized in ## conclusions and future work attention based transformer models @xcite have achieved great success in all areas of nlp @xcite @xcite . transformer models retain an autoencoderlike parallel with encoder and decoder modules. though they are dissimilar to vaes in the sense that often, decoder-only models are used for text generation, whereas encoder-only models are used for representation learning, using both simultaneously creates a vae-like architecture. this analogy has been used to train massive vaes for nlp e.g. optimus @xcite ) that employ pretrained encoders. we view our work as an indication that deterministic autoencoding, unlike traditional vaes, can design better autoencoders for text. issues of vae training, namely posterior collapse, worsen with increasing model power @xcite @xcite . powerful vae design requires tackling this and deterministic models may offer the solution. we also consider our work to be of significance to the field in its successful usage of contrastive learning for nlp, which for text often suffers from less clear augmentations @xcite relative to images. we focus on batchnorm, however, in natural language tasks, increasing emphasis is being laid on layer normalization aka lay-ernorm @xcite which forms a key part of transformers. we discuss the layer norm case further in the appendix.