Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation (2022)

Abstract: Text autoencoders are often used for unsupervised conditional text generation by applying mappings in the latent space to change attributes to the desired values. Recently, Mai et al. (2020) proposed Emb2Emb, a method to learn these mappings in the embedding space of an autoencoder. However, their method is restricted to autoencoders with a single-vector embedding, which limits how much information can be retained. We address this issue by extending their method to Bag-of-Vectors Autoencoders (BoV-AEs), which encode the text into a variable-size bag of vectors that grows with the size of the text, as in attention-based models. This allows to encode and reconstruct much longer texts than standard autoencoders. Analogous to conventional autoencoders, we propose regularization techniques that facilitate learning meaningful operations in the latent space. Finally, we adapt Emb2Emb for a training scheme that learns to map an input bag to an output bag, including a novel loss function and neural architecture. Our empirical evaluations on unsupervised sentiment transfer show that our method performs substantially better than a standard autoencoder.

## introduction in conditional text generation, we would like to produce an output text given an input text. hence, parallel input-output pairs are required to train a good supervised machine learning model on this type of task. large-scale pretraining @xcite @xcite can alleviate the necessity for training examples to some extent, but even this requires a substantial number of annotations @xcite . this is an expensive process and can introduce unwanted artifacts itself, which are henceforth learned by the model @xcite . for these reasons, there is substantial interest in unsupervised solutions. text autoencoders (aes) don't require labeled data for training, and are therefore a popular model for unsupervised approaches to many tasks, such as machine translation @xcite , sentence compression @xcite and sentiment transfer @xcite . the classical text ae @xcite embeds the input text into a single fixed-size vector via the encoder, and then tries to reconstruct the input text from the single vector via the decoder. single-vector embeddings are very useful, because they allow to perform conditional text generation through simple mappings in the embedding space, e.g. by adding a constant offset vector to change attributes such as sentiment @xcite . recently, @xcite proposed emb2emb, a method that can learn these mappings directly in the embedding space of any pretrained single-vector ae. this is a powerful framework, because the ae can then be pretrained on unlimited amounts of unlabeled data before applying it to any downstream application. this concept, transfer learning, is arguably one of the most important drivers of progress in machine learning in the recent decade: these so-called foundation models @xcite have revolutionized natural language understanding (e.g, bert @xcite ) and computer vision (e.g, dall-e @xcite ), among others. since emb2emb was designed to work with any pretrained ae, it was an important step towards their scalability. however, as @xcite point out, another crucial model property is expressivity, the ability to represent the data distribution it is trained on. in this regard, single-vector representations are fundamentally limited; they act as a bottleneck, causing the model to increasingly struggle to encode longer text @xcite . in this paper, we extend conditional text generation methods from single-vector bottleneck aes to bag-of-vector autoencoders (bov-aes), which encode text into a variable-size representation where the number of vectors grows with the length of the text. this gives bov-aes the same kind of representations as attention-based models. but this added expressivity comes with additional challenges: first, it can more easily overfit, leading to a non-smooth embedding space that is difficult to learn in. secondly, as illustrated in figure 1 , in the single-vector case, an operation φ in the vector space consists of a simple vector-to-vector mapping, and a single-vector loss. but with bov-aes, φ needs to map a bag of vectors onto another bag of vectors, for which the single-vector mapping and loss are not applicable. in this paper, we demonstrate how such a mapping can be learned in the context of the emb2emb framework by making the following novel contributions: (i) we propose a regularization scheme for bov-aes, (ii) a neural mapping architecture φ for emb2emb, and (iii) a suitable training loss. empirically, we show on two unsupervised sentiment transfer datasets @xcite of drastically different text lengths that bov-aes perform substantially better than standard aes if the text is too long to be captured by one vector alone. our ablation studies confirm that our technical contributions are crucial for this success. in the following section, we review the emb2emb framework, before we introduce bov-ae (section 3) and its integration within emb2emb (section 4). ## background: emb2emb embedding-to-embedding (emb2emb) @xcite is a general framework for both supervised and an unsupervised conditional text generation. the core idea is to disentangle the specific task from the transition from the discrete text space to a continuous latent space (plug and play), allowing for larger-scale pretraining with unlabeled data. the workflow of the framework is depicted in figure 2 . first, a text ae @xmath0: @xmath1, and back to x via a decoder dec : @xmath2, such that a(x) = x, typically trained via negative log-likelihood, l @xmath3, x). in contrast to other methods, a can in principle be any ae, opening the possibility for large-scale ae pretraining with unlabeled data. second, taskspecific training is performed only in the embedding space z of the ae. to this end, the encoder is frozen, and a new mapping layer φ : @xmath4, which is trained to transform the embedding of the input z x into the embedding of the predicted output ẑy . the concrete loss l(ẑ y ) depends on the type of task. in the supervised case, the true output is also encoded into space z, and the distance between the true embedding and the predicted embedding is minimized. in the unsupervised case, the loss needs to be defined for the specific task at hand. for example, for sentiment transfer, where the goal is to transform a negative review into a positive review while retaining as much of the input as possible, @xcite compose the loss as a combination of two loss terms foot_0 , l(ẑ y ) = l sim (z x , ẑy )+λ sty l sty (ẑ y ). l sty encourages ẑy to be classified as a positive review according to a separately trained sentiment classifier. l sim encourages the output to be close to the input in embedding space, e.g. via euclidean distance. λ sty is a hyperparameter that controls the importance of changing the sentiment of the predicted output. a main question in emb2emb is how to choose the embedding space z. mai et al. (2020) use a single continuous vector to encode all the information of the input, i.e. @xmath5. this choice simplifies the mapping φ to an mlp and the training loss to vector space distances, which is relatively easy to train. on the other hand, it limits the model in fundamental ways: the representation is fixed-sized, i.e., the representation cannot grow in size. sequence-to-sequence models with a fixed-size bottleneck struggle to encode long text sequences @xcite , which is a key reason why attention-based models are now standard practice in sequence-to-sequence models. hence, it would be desirable to adapt emb2emb in such a way that z contains variable-sized embeddings instead. ## bag-of-vectors autoencoder we propose bag-of-vectors autoencoders (bov-aes) which facilitate learning mappings in the embedding space. following the naming convention by @xcite , we refer to a bag of vectors as a (multi)-set of vectors that (i) can grow arbitrarily large, and (ii) where the elements are not ordered (a basic property of sets). a type of bov representation that is used very commonly is found in transformer @xcite enc dec the food was terrible the food was amazing ... enc dec the food was terrible the food was amazing ... enc dec text autoencoder pretraining inference enc dec enc task training figure 2: high-level view of the emb2emb framework. text autoencoder pretraining: an autoencoder is trained on an unlabeled corpus, i.e., the encoder enc transforms an input text x into a continuous embedding z x , which is in turn used by the decoder dec to predict a reconstruction x of the input sentence. task training: the encoder is frozen (grey), and a mapping φ is trained (green) on input embeddings z x to output predictions ẑy such that it minimize some loss l(ẑ y ). inference: to obtain textual predictions ŷ, the encoder is composed with φ and the decoder. ## emb2emb with bov-aes in the following we describe how to adapt the emb2emb model to bov-aes, i.e., how to generate an output bag @xmath6, . . . , ẑn } given an input bag x through the mapping φ(x), and how to choose the loss function l( x, x). for example, in the case of style transfer, we want x to be similar to x. ## experiments our experiments are designed to test the following two hypotheses. h1: if the input text is too long to be encoded into a fixed-size single vector representation, bov-ae-based emb2emb provides a substantial advantage over the fixed-sized model. h2: our technical contributions, namely l0drop regularization, the training loss, and the mapping architecture, are necessary for bov-ae's success. we evaluate our model on two unsupervised conditional text generation tasks: in section 5.1, we show that h1 holds even when the single-vector dimensionality is large (@xmath7. to this end, we create a new sentiment transfer dataset, yelp-reviews, whose inputs are relatively long. however, training on this dataset is computationally very demanding foot_1 . therefore, we turn to a short-text style transfer dataset to test hypothesis h2 (section 5.2). additionally, we conducted experiments on abstractive sentence summarization @xcite . these provide evidence of the generality of our method, as well as the utility of the mapping architecture's copy mechanism. due to space constraints, these are included in appendix a. for each of the experiments in this section, we provide full experimental details in appendix b. evaluation metrics: in sentiment transfer, the goal is to rewrite a negative review as a positive review while keeping as much of the content as possible. hence, two metrics are important, sentiment transfer ability and content retention. following common practice @xcite @xcite , we measure the former with a separately trained style classifier based on distilbert @xcite , and content retention in terms of self-bleu @xcite between the input and the predicted output. to allow comparison via a single score, we aggregate content retention and transfer accuracy @xcite , per sentence @xcite , and compute a single @xmath8, x) where x is the input sentence, ŷ is the predicted sentence, and m is the number of data points. for readability, we multiply all metrics by 100 before reporting. autoencoder pretraining: since emb2emb is plug and play, the autoencoder pretraining can be decoupled from the downstream task, enabling large-scale pretraining on a general purpose corpus. while this would certainly be necessary to reach the best results possible, such an endeavor is very resource-intensive, making it impractical to conduct the kind of controlled experiments needed to support our hypotheses. moreover, existing pretrained autoencoders such as bart @xcite ) cannot be used off-the-shelf because they weren't trained to have a smooth embedding space, for example using l0drop. in appendix c.2.2, we study the effect of adding an l0drop layer inside bart and finetuning it for a few steps on the target task data. although this works to some extent, this l0drop layer can be expected to remove information which would be kept if it were trained in full large-scale pretraining, which we don't have the resources to do. therefore, we instead pretrain all autoencoders from scratch directly on the data of the target task. models named l0-r denote l0drop-based bov-ae models that only differ in the target ratio r used in training. as a control, we always compare to a single vector fixed-size ae, which is obtained by averaging the vectors at the last layer of the encoder.