Sentence Bottleneck Autoencoders from Transformer Language Models (2021)

Abstract: Representation learning for text via pretraining a language model on a large corpus has become a standard starting point for building NLP systems. This approach stands in contrast to autoencoders, also trained on raw text, but with the objective of learning to encode each input as a vector that allows full reconstruction. Autoencoders are attractive because of their latent space structure and generative properties. We therefore explore the construction of a sentence-level autoencoder from a pretrained, frozen transformer language model. We adapt the masked language modeling objective as a generative, denoising one, while only training a sentence bottleneck and a single-layer modified transformer decoder. We demonstrate that the sentence representations discovered by our model achieve better quality than previous methods that extract representations from pretrained transformers on text similarity tasks, style transfer (an example of controlled generation), and single-sentence classification tasks in the GLUE benchmark, while using fewer parameters than large pretrained models.

## introduction recent research has focused on devising new unsupervised pretraining methods from unlabeled data that involves some form of language modeling, primarily autoregressive @xcite , masked @xcite @xcite and generalized @xcite @xcite , with much success on downstream tasks. under the hood, most of these methods use transformers @xcite for encoding text sequences, which allows them to learn powerful contextual word representations that have been used widely for building models in nlp. however, this does not hold for sentence representations derived from pretrained transformer language models based on a special token or basic pooling operations. to this end, representation learning methods have been designed to better capture semantic information from pretrained transformer language models, e.g., using siamese networks trained with a triplet loss @xcite or transforming the desired sentence distribution to a gaussian distribution through normalizing flows @xcite . existing sentence representations directly derived from pretrained language models or learned by specialized methods cannot guarantee perfect reconstruction of the input, a property that can enhance the structure of their semantic space and enable their use for controlled generation tasks. for the latter, a few recent studies have looked into ways to steer generation of pretrained language models towards a particular style @xcite , although they require following the gradient during the sampling process and rely on style text classifiers which might not be always available. the latent space of a text autoencoder allows one to perform controlled text generation by directly manipulating sentence representations using basic numerical operations @xcite . yet, how to convert pretrained transformer language models to autoencoders with such properties still remains unexplored. to fill in this gap, we introduce autobot, a new autoencoder model for learning sentence "bottleneck" (i.e., fixed-size) representations from pretrained transformers that is useful for similarity, generation, and classification, displayed in figure 1 . our model has two unique components: (i) a transformation that uses dot product attention to dynamically pool semantic information from the pretrained model's hidden states into a sentence bottleneck representation, and (ii) a shallow transformer decoder that is modified to operate based on the bottleneck representation. instead of training our autoencoder from scratch, we directly finetune it using an input reconstruction objective on the unlabeled data on which the original pretrained transformer was trained. we keep the underlying pretrained transformer encoder fixed, which makes it more efficient than training from scratch and proves beneficial even if we compare to pretrained transformers trained for an equal number of steps. our evaluation on representative sentence similarity, classification, and generation tasks demonstrates that the resulting sentence representations are compact, better capture semantic similarity at the sentence-level than strong sentence representation methods @xcite , and can be used for controlled generation tasks. lastly, our model performs almost on par with the large roberta model @xcite even though it only introduces 1.6% additional parameters relative to the base roberta model. ## model: autobot taking inspiration from recent research on text autoencoders @xcite @xcite , we extend standard autoregressive text autoencoders, which have been predominantly based on recurrent networks, to a transformer-based architecture and integrate them with pretrained language models; here we focus on roberta @xcite . autoencoders generally follow the encoderdecoder model structure to reconstruct their input with the constraint that the encoder produces a single, fixed-length hidden representation enc(x) = z: here, we focus on denoising autoencoders that aim to reconstruct a perturbed version of the input @xcite , which is compatible with many of the pretrained language models that are based on masked language modeling. in our experiments, we use the same masking procedure as @xcite to perturb the input. ## experiments to assess the quality of the sentence representations learned by our model we evaluate on sentence similarity (section 3.2), classification (section 3.3), and generation tasks (section 3.4). ## related work reconstructing text with autoencoders is an active area of research that has lead to several advancements such as denoising @xcite @xcite , adversarial @xcite , and regularized @xcite ) autoencoders. they have been found especially useful in controlled text generation @xcite @xcite , especially in sentiment style transfer @xcite . the encoder-decoder structure for obtaining representations has been used in pretraining @xcite , sentence infilling @xcite , and multilingual (artetxe and schwenk, 2019) scenarios. in particular, @xcite treat denoising as translation task to perform pretraining from scratch, but their approach does not induce a sentence representation space with generative properties. in contrast, our method makes use of a frozen pretrained transformer to learn a shallow, sentence bottleneck autoencoder on top. ## conclusion we proposed an approach that converts a pretrained transformer language model into a sentence-level autoencoder that is able to reconstruct its pretraining data. the resulting model improves the performance of the pretrained model on sentence-level tasks while maintaining its performance on multisentence tasks. in addition, the new sentence representations are suitable for efficient conditional text generation such as sentiment transfer without the need for training on in-domain data. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, ł ukasz kaiser, and illia polosukhin. 2017. attention is all you need. in i. guyon, u. v. luxburg, s. bengio, h. wallach, r. fergus, s. vishwanathan, and r. garnett, editors, advances in neural information processing systems 30, pages 5998-6008. curran associates, inc. pascal vincent, hugo larochelle, isabelle lajoie, yoshua bengio, pierre-antoine manzagol, and léon bottou. 2010. stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. journal of machine learning research, 11(12). alex wang, amanpreet singh, julian michael, felix hill, omer levy, and samuel bowman. 2018. glue: a multi-task benchmark and analysis platform for natural language understanding. in proceedings of the 2018 emnlp workshop blackboxnlp: analyzing and interpreting neural networks for nlp, pages 353-355. ke wang, hang hua, and xiaojun wan. 2019. controllable unsupervised text attribute transfer via editing entangled latent representation. in advances in neural information processing systems, pages 11034-11044. junbo jake zhao, yoon kim, kelly zhang, alexander m rush, yann lecun, et al. 2018. adversarially regularized autoencoders. in proceedings of the 35th international conference on machine learning, proceedings of machine learning research. pmlr. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of the ieee international conference on computer vision, pages 19-27.