Cycle-Consistent Adversarial Autoencoders for Unsupervised Text Style Transfer (2020)

Abstract: Unsupervised text style transfer is full of challenges due to the lack of parallel data and difficulties in content preservation. In this paper, we propose a novel neural approach to unsupervised text style transfer which we refer to as Cycle-consistent Adversarial autoEncoders (CAE) trained from non-parallel data. CAE consists of three essential components: (1) LSTM autoencoders that encode a text in one style into its latent representation and decode an encoded representation into its original text or a transferred representation into a style-transferred text, (2) adversarial style transfer networks that use an adversarially trained generator to transform a latent representation in one style into a representation in another style, and (3) a cycle-consistent constraint that enhances the capacity of the adversarial style transfer networks in content preservation. The entire CAE with these three components can be trained end-to-end. Extensive experiments and in-depth analyses on two widely-used public datasets consistently validate the effectiveness of proposed CAE in both style transfer and content preservation against several strong baselines in terms of four automatic evaluation metrics and human evaluation.

## introduction unsupervised text style transfer is to rewrite a text in one style into a text in another style while the content of the text remains the same as much as possible without using any parallel data. style transfer can be utilized in many tasks such as personalization in dialogue systems @xcite , sentiment and word decipherment @xcite , offensive language translation @xcite , and data augmentation @xcite @xcite , etc. however, there are a variety of challenges to text style transfer in practice. first, we do not have large-scale style-to-style parallel data to train a text style transfer model in a supervised way. second, even with non-parallel corpora, the inherent discrete structure of text sequences aggravates the difficulty of learning desirable continuous representations for style transfer @xcite @xcite . third, it is difficult to preserve the content of a text when its style is transferred. to obtain good content preservation for text style transfer, various disentanglement approaches @xcite @xcite are proposed to separate the content and style of a text in the latent space. however, content-style disentanglement is not easily achievable as content and style typically interact with each other in texts in subtle ways @xcite . in order to solve the issues above, we propose a cycle-consistent adversarial autoencoders (cae) for unsupervised text style transfer. in cae, we learn the representation of a text where we embed both content and style in the same space. such space is constructed for each style from non-parallel data. we then transfer the learned representation from one style space to another space. to guarantee that the content is preserved during the style transfer procedure, the transferred representation is transferred back to the original space to minimize the distance between its original representation and the reversely transferred representation. figure 1 demonstrates the diagram of the proposed cae. without loss of generality, we discuss cae for text transfer between two styles. multiple styles can be factorized into two styles @xcite . specifically, cae is composed of three essential components: lstm autoencoders, adversarial style transfer networks and a cycle-consistent constraint. the lstm autoencoder contains an encoder enc to encode a sentence x s i from style s into a hidden representation z s i in the corresponding style space and a decoder dec to generate sentences from vectors z s i learned by the lstm encoder (or zj transferred from the other style space). the adversarial style transfer networks learn a generator t to generate a representation zs 1 →s 2 i in style space s 2 from z s 1 i in style space s 1 , or the other way around from style space s 2 to s 1 . it also uses a discriminator to ensure that the transferred representations belong to the corresponding style space. the top of figure 2 displays the original sentences and style-transferred sentences generated by the lstm decoder from transferred representations produced by the generator of the adversarial style transfer network. the cycle-consistent constraint transfers back representations to their original space and attempts to minimize their distances, as demonstrated in the bottom of figure 2 . in summary, our contributions are threefold as follows. ## related work a number of text style transfer approaches have been proposed in recent years following the pioneering study of style transfer in images @xcite . these approaches can be roughly categorized into two strands: methods that disentangle representations of style and content and the others that do not. in the first line of text style transfer, @xcite combine a variational autoencoder (vae) with style discriminators to enforce that styles can be reliably inferred back from generated sentences. @xcite uses discriminators to align hidden states of the transferred samples from one style with the creativecommons.org/licenses/by/4.0/. figure 2 : the visualization of style transfer and cycle-consistent constraint in cae. upper: sentences in one style and style-transferred sentences in another style. bottom: cycle-consistent constraint enforcing that sentences transferred into a different style can be translated back to its original meaning in its original style. in the second line of works that avoid disentangled representations of style and content, @xcite use back-translation technique on denoising autoencoder model with latent representation pooling to control the content preservation. their experiments and analyses show that the contentstyle disentanglement is neither necessary nor always satisfied with practical requirements, even with the domain adversarial training that explicitly aims at learning disentangled representations. style transformer @xcite uses transformer as a basic module to train a style transfer system. dualrl @xcite employs a dual reinforcement learning framework with two sequence-to-sequence models in two directions, using style classifier and back-transfer reconstruction probability as rewards. we follow the second line and propose a novel method that makes no assumption on the latent representation disentanglement. but differently, we perform style transfer in the latent representational spaces of the source and target style. and inspired by cyclegan @xcite which uses a cycle loss on image style transfer to enforce the back-translation of a transferred image to be equivalent to the original image, we also impose a cycle-consistent constraint on our style transfer network. however, training style transfer networks with such a cycle constraint on discrete texts is quite different from those on images and non-trivial. in order to enable cycle training on texts, we project texts onto adversarially regularized latent space collectively learned by the lstm autoencoders and adversarial transfer networks. different from latent cross project with euclidean distance for semi-supervised style transfer @xcite , we construct latent cyclegan to generate high quality sentences for unsupervised style transfer. ## cae: cycle-consistent adversarial autoencoders suppose we have two non-parallel text datasets x 1 = {x 1 i } n @xmath0. the cae employs lstm autoencoder models to encode discrete text sequences x 1 i , x 2 j into representations z 1 i , z 2 j , and to generate sentences @xmath1, @xmath2, @xmath3, t 2→1 from z 1 i , z 2 j . ## experiments to compare our work with previous approaches to text style transfer from non-parallel data, we conducted experiments on two text transfer tasks: sentiment transfer on the yelp restaurant review corpus and topic transfer on the "yahoo! answers comprehensive questions and answers version 1.0" dataset. we also carried out ablation experiments to study the impact of different components of cae on overall performance of style transfer. ## conclusion we have presented a novel approach, cae, to unsupervised text style transfer from non-parallel text. we learn latent representations for sequences in different styles with lstm autoencoders. the learned representations are transferred from their original style to another style via adversarial transfer networks. the transfer networks are equipped with a cycle-consistent constraint to guarantee content preservation during style transfer. experiments and analyses on the yelp and yahoo datasets sufficiently demonstrate the powerful style transfer ability of cae with good fluency and content preservation against previous methods.