So Different Yet So Alike! Constrained Unsupervised Text Style Transfer (2022)

Abstract: Automatic transfer of text between domains has become popular in recent times. One of its aims is to preserve the semantic content while adapting to the target domain. However, it does not explicitly maintain other attributes between the source and translated text: e.g., text length and descriptiveness. Maintaining constraints in transfer has several downstream applications, including data augmentation and debiasing. We introduce a method for such constrained unsupervised text style transfer by introducing two complementary losses to the generative adversarial network (GAN) family of models. Unlike the competing losses used in GANs, we introduce cooperative losses where the discriminator and the generator cooperate and reduce the same loss. The first is a contrastive loss and the second is a classification loss — aiming to regularize the latent space further and bring similar sentences closer together. We demonstrate that such training retains lexical, syntactic and domain-specific constraints between domains for multiple benchmark datasets, including ones where more than one attribute change. We show that the complementary cooperative losses improve text quality, according to both automated and human evaluation measures.

## introduction modern neural networks methods are capable of mapping data from one domain to another. prominent examples include translation of text between languages @xcite @xcite , emoji creation from human faces @xcite , and stylistic transfer of speech @xcite . in natural language processing (nlp), the umbrella term attribute transfer @xcite (or domain i really loved murakami's book personal pronoun proper noun loved the movie text style transfer smaller length no personal pronoun no proper noun i absolutely enjoyed spielberg's direction text style transfer + constraints similar length similar personal pronoun domain appropriate proper noun vs. transfer) refers to similar methods foot_0 . the aim is to maximally preserve the semantics of the source sentence ("content") but change other properties ("attributes"), such as sentiment @xcite , expertise @xcite , formality @xcite or a combination of them @xcite . text style transfer, a popular form of attribute transfer, regards "style" as any attribute that changes between datasets @xcite . building on the progress of supervised transfer models, recent works have focused on unsupervised style transfer that avoids costly annotation of parallel sentences. however, models built using unsupervised methods perform poorly when compared to supervised (parallel) training @xcite . these methods, while capable of achieving the target domain characteristics, often fail to maintain the invariant content. figure 1 illustrates one such example, where a sentence from the books domain is translated to the movie domain. while the translated sentence "loved the movie" has correctly transferred the attribute (style), it does not have the same length, does not retain the personal noun ("i"), nor use a domain-appropriate proper noun. comparatively, the higher-fidelity transfer "i absolutely enjoyed spielberg's direction", maintains such constraints of identity, in addition to being apt. this problem setting is an important application of text transfer, as enforcing constraints of identity can help maintain the brand identity when the product descriptions are mapped from one commercial product to another. they can also help in data augmentation for downstream domain adaptation nlp applications ( § 5). constraints of identity are explored extensively in the computer vision task of cross-domain image generation. @xcite , but these issues-to the best of our knowledge-are unexplored in nlp. in this paper, we improve unsupervised attribute transfer by enforcing invariances via explicit constraints. current methods in text attribute transfer lack mechanisms to explicitly enforce such constraints between the source and the transferred sentence. to this end, we build upon unsupervised text style transfer work by introducing an additional explicit regularization component in the latent space of a gan-based seq2seq network through two complementary losses ( § 3). unlike the adversarial losses in the gan framework, our proposed losses cooperatively reduce the same objective. the first loss is a contrastive loss @xcite that brings sentences that have similar constraints closer and pushes sentences that are dissimilar farther away. the second loss is a classification loss that helps maintain the sentence identity via constraints from the latent vectors @xcite . our approach, while simple and aimed at maintaining constraints, improves the overall performance of the generation. we demonstrate these gains over three datasets: yelp @xcite , imdb @xcite and political @xcite , generating six constraints including lexical, syntactic and domainspecific constraints. the introduced cooperative losses satisfy the constraints more effectively compared against strong baselines. since multiple attributes can change between two domains @xcite , we test our method on one such dataset and show that the constraints of identity are maintained more effectively ( § 4.4.2). to the best of our knowledge, our approach is the first to introduce cooperative losses in a gan-like setup for nlg. ## preliminaries task setup: we consider two sets of sentences (or corpora) @xmath0, x 2 src , . . . x m src } and @xmath1, x 2 trg , . . . x n trg }, as the source and target domains, respectively. each corpus -which we interpret as domains -contain discernable attributes, ranging from sentiment (e.g., positive vs. negative), topics, political slant (e.g., democratic vs. republican), or some combination @xcite . the overall task is to rewrite a piece of text s @xmath2, such that the translation changes the attributes varying across the two domains but retains the remaining content. while content retention is not explicitly defined in the literature, we design this new task of constrained unsupervised attribute transfer that assigns explicit constraints @xmath3, c 2 , . . . , c |c| }, to be retained. these constraints can be defined at various levels of a sentence: lexical, syntactic and domain-specific. ## experiments datasets. we use three datasets with single attribute changes: i) yelp reviews: business reviews listed on yelp, labeled as either a positive or negative sentiment. ii) imdb movie reviews: consists of movie reviews @xcite 2019) we provide a summary of the dataset statistics in constraints: we constrain every sentence along six diverse dimensions that we desire to control ## discussion and limitations cycle consistency loss: a) in latent spaces -cycle consistency in latent spaces has been shown to improve word-level tasks, such as cross-lingual dictionary construction @xcite and topic modeling @xcite . a recent work from @xcite claims to improve unsupervised style transfer using such losses. in our experiments, however, it did not result in any noticeable performance improvement foot_2 . given this, we hypothesize that cycle consistency might be too restrictive for sentence-level tasks. b) using back-translation-back-translation is another alternative to ensure semantic consistency between source and the target sentence @xcite @xcite . however, in our case, since we are training an arae, it would involve an additional inference and auto-encoder training step which is expensive and we defer exploring this. using transformers: we also replace our lstm auto-encoders with both pre-trained and randomly initialized transformer encoder-decoders @xcite . although we found an increase in the agg, it was mostly because of very high sim and very low acc. reducing the number of layers, attention heads would still result in a large model that is still prone to copying text. this reveals the potential limitations of our method and training using transformers is a future work. transferred sentences as adversarial examples: we demonstrate an important application of our proposed constrained transfer by considering them as adversarial examples for domain adaptation. domain adversarial neural network (dann) @xcite is an unsupervised domain adaptation method that improves performance of an end-task (e.g, sentiment analysis) on a target domain considering only supervised data from source domain. we train dann for sentiment analysis on amazon reviews dataset @xcite with dvd as source and electronics as the tar-get domain -achieving an accuracy of 83.75% on electronics. next, we train the best variant of arae seq2seq to transfer a separate set dvd reviews to electron-ics reviews and use them as adversarial examples to test the dann model 6 . we find that the accuracy of dann on the electronics domain reduces by ∼3 points. this shows the potential application of domain transferred sentences as adversarial examples. similar ideas have been tried for image style transfer @xcite , but needs more investigation in nlp. ## related work text attribute transfer has a vast literature @xcite with deep learning methods becoming popular. the methods are either supervised (requiring parallel data) or unsupervised. supervised methods re-purpose sequence to sequence models used in machine translation to achieve the goals @xcite . however, obtaining parallel data is cumbersome and thus unsupervised methods that consider pseudo-parallel data have become popular. disentanglement approaches are the prevalent approach to tackle unsupervised attribute transfer: attributes and content are separated in latent dimension. to disentangle the attributes adversarial methods maximize the loss of a pre-trained attribute classifier @xcite @xcite . however, the literature has paid little attention in defining and preserving content. cycle consistency losses -imposing that reconstruction from the target style sentence should resemble the source sentence -is the most prevalent @xcite @xcite @xcite . however, this is expensive and nondifferentiable, thus requiring reinforcement learning techniques to enforce it. our work defines the different constraints that should be preserved and adds simple differentiable contrastive learning losses to preserve them. in recent times, text style transfer models are moving away from disentanglement approaches @xcite . recent works that use transformers for style transfer also have adopted this @xcite . how6 since each of dvd and electronics contain positive and negative reviews, we test whether transferred sentences maintain the appropriate sentiment and find the accuracy to be 79%. ever, these methods do not explicitly maintain the constraints between the two styles which is the main aim of our work. ## conclusion text style transfer works focuses on retaining content and changing the style of sentences but does not maintain other desirable constraints. we address this by introducing two cooperative losses to the gan-inspired adversarially regularized autoencoder (arae) that further regularizes the latent space. while satisfying the constraints our methods brings significant improvements in overall score. while we focus on simple constraints at the sentence-and word-level, future work can add phrase-level and more fine-grained constraints. potential future work may explore reinforcement learning losses to directly optimize the constraints.