On Learning Text Style Transfer with Direct Rewards (2021)

Abstract: In most cases, the lack of parallel corpora makes it impossible to directly train supervised models for the text style transfer task. In this paper, we explore training algorithms that instead optimize reward functions that explicitly consider different aspects of the style-transferred outputs. In particular, we leverage semantic similarity metrics originally used for fine-tuning neural machine translation models to explicitly assess the preservation of content between system outputs and input texts. We also investigate the potential weaknesses of the existing automatic metrics and propose efficient strategies of using these metrics for training. The experimental results show that our model provides significant gains in both automatic and human evaluation over strong baselines, indicating the effectiveness of our proposed methods and training strategies.

## introduction text style transfer aims to convert an input text into another generated text with a different style but the same basic semantics as the input. one major challenge in this setting is that many style transfer tasks lack parallel corpora, since the absence of human references makes it impossible to train the text style transfer models using maximum likelihood estimation (mle), which aims to maximize the predicted likelihood of the references. as a result, some of the earliest work @xcite @xcite on unsupervised text style transfer proposed training algorithms that are still based on mle by formulating the style transfer models as auto-encoders optimized with reconstruction loss. specifically, during training the model is tasked to generate a style-agnostic encoding and reconstruct the input text based on this encoding with style-specific embeddings or decoders. during inference, the model aims to transfer the source 1 code and data are available at: https://github. com/yixinl7/direct-style-transfer text style using the target style information. while these methods have seen empirical success, they face the inherent difficulty of coming up with a style-agnostic but content-preserving encodingthis is a non-trivial task and failure at this first step will diminish style transfer accuracy and content preservation of the final output. another line of work @xcite @xcite proposes training algorithms based on rewards related to the automatic evaluation metrics, which can assess the model performance more directly during training. this approach is conceptually similar to training algorithms that optimize models using rewards related to the corresponding evaluation metrics for other nlp tasks, such as machine translation @xcite ) or text summarization @xcite . as for unsupervised style transfer, the widely used automatic metrics mainly attend to three desiderata: (1) style transfer accuracy -the generated sentence must be in the target style, commonly measured by the accuracy of a style classifier applied to the transferred text, (2) fluency -the generated text must be grammatically correct and natural, commonly measured by the perplexity of a language model and (3) content preservation -the semantics need to be preserved between the source and target, commonly measured by the bleu score between the system outputs and source texts. since these automatic metrics only require the system outputs and source texts, they can be used as rewards for training. moreover, the two lines of approaches can be used together, and previous work @xcite @xcite proposed methods which use the auto-encoders as the backbone augmented with task-specific rewards. in particular, the style transfer accuracy reward is used by most of the recent work. however, reward-based training algorithms still have their limitations, and in this paper we aim to identify and address the bottlenecks of these methods. specifically, we focus on two problems: (1) the difficulty of designing an efficient reward for content preservation, (2) the lack of robustness of the existing automatic evaluation metrics. content preservation is more difficult to measure compared to style transfer accuracy and fluency because it needs to consider the overlap in the semantics between the source text and system outputs. while using bleu score between the source text and system output would be a direct solution @xcite , this approach has an inherent limitation in that n-gram based metrics such as bleu are sensitive to lexical differences and will penalize modifications that are necessary for transferring text style. in fact, previous work has proposed various different proxy rewards for content preservation. one of the most popular methods is the cycle-consistency loss @xcite @xcite , which introduces a round-trip generation process, where the model generates an output in the target style, and the ability of a reconstruction model to re-generate the original text is used as a proxy for content preservation. while this method is more tolerant to lexical differences, the correlation between the reconstruction loss and content preservation can be weak. therefore, we aim to design a reward for content preservation which can directly assess the semantic similarity between the system outputs and input texts. specifically, we note that models of semantic similarity are widely studied @xcite @xcite , and we can leverage these methods to directly calculate the similarity between the system outputs and input texts. this renders our method applicable for even unsupervised settings where no human references are available. another key challenge for reward-based training algorithms is that the existing automatic evaluation metrics are not well-correlated with human evaluation @xcite . it poses general risks to the work in this field with respect to model training and evaluation since these metrics are widely used. an important observation we made from our experiments is that style transfer models can exploit the weaknesses of the automatic metrics. they do this by making minimal changes to the input texts which are enough to trick the classifier used for style transfer accuracy while achieving high content preservation and fluency scores due to the high lexical similarity with the input texts. upon identifying this risk, we re-visit and propose several strategies that serve as auxiliary regularization on the style transfer models, effectively mitigating the problem discussed above. we empirically show that our proposed reward functions can provide significant gains in both automatic and human evaluation over strong baselines from the literature. in addition, the problems we identify with existing automatic evaluation metrics suggest that the automatic metrics need to be used with caution either for model training or evaluation in order to make it truthfully reflect human evaluation. ## analysis we next show an ablation study, demonstrating the effectiveness of the content preservation and fluency rewards in dirr, and how sim can be used to replace the cycle-consistency loss. we also compare using bleu versus using sim as a content-preservation reward, finding that using bleu results in reduced performance, unstable training, and artifacts in the outputs, which makes the results less natural than the results of the model trained with sim score. to illustrate that training with sim can replace the cycle-consistency loss for content preservation, we fine-tuned dirr-cycle on sim to produce a new model, dirr w/o flu. the difference between dirr and dirr w/o flu is that the former is additionally trained with our fluency rewards. the results are shown in ## related work a main line of work @xcite @xcite @xcite ) for text style transfer aims to model the conditional distribution of the data with the encoderdecoder architecture. due to the lack of parallel corpora, inductive biases are designed to make the generation conditioned on both source sentences and specific styles such that the model can rewrite the source texts with the target style while still preserve the content information of the source texts. efforts are also made to design training objectives to improve performance. for example, backtranslation @xcite , denoising auto-encoding @xcite and the cycle-consistency loss @xcite @xcite have been shown effective for improving the model performance. @xcite proposes a retrievebased pipeline, which contains three stages, namely, delete, retrieve and generate. @xcite extends this pipeline by using gpt @xcite as the generator. compared to these methods, we propose a more direct and effective approach to encourage semantic-preserving transfer by directly measuring the semantic similarity of the source texts and system outputs. recently, other works have been proposed for unsupervised text style transfer @xcite @xcite . @xcite proposes a probabilistic view which models the non-parallel data from two domains as a partially observed parallel corpus. @xcite proposes a tag-and-generate pipeline, which firstly identifies style attribute markers from the source texts, then replaces them with a special token, and generates the outputs based on the tagged sentences. @xcite focuses on exploring the word-level style relevance which is assigned by a pre-trained style classifier. they propose a reward for content preservation which is based on the weighted combination of the word embeddings of the source texts and system outputs. compared to this reward, our proposed content reward is specifically designed for semantic similarity and pre-trained on large corpora, which makes it more robust across different datasets. ## conclusion in this paper, we propose a direct approach of improving content preservation for text style transfer by leveraging a semantic similarity metric as the content reward. using a large pre-trained language model (gpt-2) with our proposed rewards that target the different aspects of the output quality, our approach achieves strong performance on both automatic and human evaluation. recently, several semantic similarity metrics @xcite @xcite based on pre-trained language models have shown promising results. introducing these metrics in our proposed method as the content preservation reward may bring further improvements. moreover, we identify several problems in the commonly used automatic evaluation metrics and datasets, and propose several practical strategies to mitigate these problems, which makes these met-rics more effective rewards for model training. considering the weaknesses of the automatic metrics presented in this work, we believe that more rigorous discussion and investigation on the criteria of "successful transferring" is essential for this field of work. since existing works mostly relied on model-based metrics to determine the success of style transfer models, methods such as adversarial training could be introduced to make the modelbased metrics more robust and faithful indicators of the success of style-transferring, which would be beneficial for both model training and evaluation.