Plug and Play Autoencoders for Conditional Text Generation (2020)

Abstract: Text autoencoders are commonly used for conditional generation tasks such as style transfer. We propose methods which are plug and play, where any pretrained autoencoder can be used, and only require learning a mapping within the autoencoder’s embedding space, training embedding-to-embedding (Emb2Emb). This reduces the need for labeled training data for the task and makes the training procedure more efficient. Crucial to the success of this method is a loss term for keeping the mapped embedding on the manifold of the autoencoder and a mapping which is trained to navigate the manifold by learning offset vectors. Evaluations on style transfer tasks both with and without sequence-to-sequence supervision show that our method performs better than or comparable to strong baselines while being up to four times faster.

## introduction conditional text generation foot_0 encompasses a large number of natural language processing tasks such as text simplification @xcite , summarization @xcite , machine translation @xcite and style transfer @xcite . when training data is available, the state of the art includes encoder-decoder models with an attention mechanism @xcite which are both extensions of the original sequence-to-sequence framework with a fixed bottleneck introduced by @xcite . despite their success, these models are costly to train and require a large amount of parallel data. yet parallel data is scarce for conditional text generation problems, necessitating unsupervised figure 1 : the manifold of a text autoencoder is the lowdimensional region of the high-dimensional embedding space where texts are actually embedded. the example shows the mapping of a source sequence x with embedding z x to z y , which is the embedding of target sequence y such that it reflects the target manifold. ## proposed framework the key idea of our framework is to reduce discrete sequence-to-sequence tasks to a continuous embedding-to-embedding regression problem. our emb2emb framework for conditional generation based on pretrained autoencoders (figure 2 ) encompasses learning sequence-to-sequence tasks both where parallel input/output sentence pairs are available ("supervised") and where they are not ("unsupervised"). given a pretrained autoencoder (left of figure 2 , section 2.1) we use its encoder to encode both input and, during supervised training, the output sequence (section 2.3). 3 we then learn a continuous mapping φ from input sequence embeddings to output sequence embeddings (center of figure 2 ). in the unsupervised case (section 2.4), the task loss reflects the objectives of the task, in our experiments consisting of two terms, one to encourage content preservation and the other to encourage style transfer. in both supervised and unsupervised cases, the task-specific loss l task is combined with an adversarial term l adv that encourages the output vector to stay on the autoencoder's manifold (see figure 1 ; section 2.5), so that the complete loss function is: at inference time (right of figure 2 ; section 2.4), the decoder from the pretrained autoencoder's decoding function is composed with φ and the encoder to generate a discrete output sentence ŷ conditioned on an input sentence x: ## experiments we conduct controlled experiments to measure the benefits of the various aspects of our approach. first, we consider a supervised sentence simplification task and compare our approach to others that use a fixed-size representational bottleneck, considering also the model's sensitivity to the strength of the adversarial loss and the use of offsetnet (section 3.1). we then turn to an unsupervised sentiment transfer task, first comparing our approach to other methods that can be considered "plug and play" and then investigating the effect of plug and play when only a little labeled data is available (section 3.2). we note that the current state of the art is based on transformers @xcite ; since our aim is to develop a general-purpose framework and our computational budget is limited, we focus on controlled testing of components rather than achieving state-of-the-art performance. autoencoder. in all our experiments, we use a one-layer lstm as encoder and decoder, respectively. we pretrain it on the text data of the target task as a denoising autoencoder (dae; @xcite with the noise function from @xcite . additional training and model details can be found in appendix a. ## conclusion in this paper, we present emb2emb, a framework that reduces conditional text generation tasks to learning in the embedding space of a pretrained autoencoder. we propose an adversarial method and a neural architecture that are crucial for our method's success by making learning stay on the manifold of the autoencoder. since our framework can be used with any pretrained autoencoder, it will benefit from large-scale pretraining in future research. the best in terms of the development score metric introduced in the training details. a.5.4 model analysis experimental setup we investigate the effect of offsetnet and the adversarial training term on our unsupervised style transfer model by measuring the self-bleu score with the input sentence and the accuracy of a separately trained bert classifier (achieving 97.8% classification accuracy) on the yelp development set. we again report the best performance among 6 exponentially increasing λ adv values for each model. to inspect the behavior of the models at varying levels of transfer, we trained and plotted one model each for λ @xmath0.1, 0.5, 0.9, 0.95, 0.99}. results. the results in figure 9 show that offset-net reaches better transfer accuracy than the mlp at comparable self-bleu scores. the performance drops significantly if the adversarial term is not used. this confirms the importance of our design decisions.