Learning from Bootstrapping and Stepwise Reinforcement Reward: A Semi-Supervised Framework for Text Style Transfer (2022)

Abstract: Text style transfer is an important task in controllable language generation. Supervised approaches have pushed performance improvement on style-oriented rewriting such as formality conversion. However, challenges remain due to the scarcity of large-scale parallel data in many domains. While unsupervised approaches do not rely on annotated sentence pairs for each style, they are often plagued with instability issues such as mode collapse or quality degradation. To take advantage of both supervised and unsupervised paradigms and tackle the challenges, in this work, we propose a semi-supervised framework for text style transfer. First, the learning process is bootstrapped with supervision guided by automatically constructed pseudo-parallel pairs using lexical and semantic-based methods. Then the model learns from unlabeled data via reinforcement rewards. Specifically, we propose to improve the sequence-to-sequence policy gradient via stepwise reward optimization, providing fine-grained learning signals and stabilizing the reinforced learning process. Experimental results show that the proposed approach achieves state-of-the-art performance on multiple datasets, and produces effective generation with as minimal as 10% of training data.

## introduction text style transfer is a task in natural language generation, which aims to automatically control certain attributes during sentence paraphrasing, such as formality, sentiment, and humor @xcite . style transfer has many practical applications, such as altering emotions of spoken utterances, removing biases in transcripts, and conveying politeness in messages @xcite . the key for a successful rewrite is to preserve the semantic content of the source sentence, while transforming it to a particular target style without sacrificing fluency and grammatical accuracy. therefore, the performance of style transfer models is commonly assessed on both style accuracy and content preservation. when large-scale annotated sentence pairs are available, training neural sequence-to-sequence models via supervised learning shows impressive generation quality @xcite . however, in many use cases, it is unfeasible to adopt supervised approaches because parallel samples are unavailable. to address data insufficiency bottlenecks, various unsupervised approaches have been proposed for text style transfer, including learning disentangled representations of style and content @xcite and adopting pairwise backtranslation @xcite . recently, reinforcement learning (rl) is introduced to develop unsupervised models such that rewards of content preservation and style conversion are used to optimize sequence generation @xcite . however, rl-based methods are often challenging to train in practice. for instance, the rewards have high variance during early stages when learning from scratch, which affects the training stability; and they cannot provide finegrained learning signals as traditional token-level maximum likelihood estimation, since they are often calculated on the entire generated sequence (de masson d' @xcite . as a result, models are prone to mode collapse and often fail to produce acceptable generations in reality. herein, we propose a semi-supervised framework for text style transfer, and optimize it on training stability and signal fineness. our semisupervised model uses a small amount of parallel data for supervised learning, and gets further improvement by learning from a large amount of unlabeled data. in contrast to prior work that often relies on human-annotated parallel pairs like @xcite , the approach we propose bootstraps the training process with automatically constructed pseudo parallel data. two pseudo pair matching methods are investigated: a lexical-based strategy, which is straightforward by calculating the token-level overlap; and a semantic-based strategy, which uses semantic similarity as criteria and would have better general potential. furthermore, to obtain fine-grained signals for the rl-based sequence-to-sequence training process, we propose a stepwise reward re-weighting strategy. this is inspired by the observation that the style transfer weights are not uniform across tokens/spans in the source sentence: some tokens weigh more during attribute-guided text style transfer @xcite . therefore, instead of using the reward (e.g., style strength scores) calculated from the entire generated sentence @xcite , we use the token-level reward. specifically, we extract attribute-related attentive scores from a pre-trained style discriminator, obtain a stepwise reward by re-weighting the sequencelevel score, and utilize it as a fine-grained signal for policy gradient back-propagation. we evaluate the proposed framework that incorporates both supervision and reward-based learning on three style transfer corpora (section 4). experiments show that our model achieves state-of-theart performance. particularly, the proposed model can produce reasonable generations with only 10% training data on the yelp and amazon corpora, and it also outperforms the supervised baselines when applying on the well-annotated gyafc dataset. ## related work neural text style transfer the aim of text style transfer is to automatically convert text to a certain style while preserving the content @xcite . it has many applications, like persona-based dialogue generation @xcite . recently, neural sequenceto-sequence architectures becomes popular for this task. when parallel data are available, supervised training with cross-entropy loss is typically applied @xcite . however, annotated data are hard to obtain in many use cases, thus learning from non-parallel corpora has become an active research area. there are two approaches: (1) disentangling style and content by learning a distinct representation for each element. for example, variational autoencoders are first used to transform a sentence into a low-dimension hidden state. then the attribute-related latent representation is extracted to guide the decoder for target style generation @xcite @xcite ; ## methodology define s as the source style and t as the target style (e.g., @xmath0, @xmath1. let d s and d t be the two datasets which are comprised of sentences in each style respectively. the style transfer system, denoted as a text encoding-decoding model g, is to generate sentences in the target style. the goal is formulated to maximize p (y|x; θ g ), where θ g are the model parameters. in our setting, we make the rewriting bidirectional, i.e. it can be used to transfer source style to target style and verse versa. in this case, an additional input @xmath2, t } is fed to g specifying the style to which the sentence is to be converted. hence, the objective is to maximize p (y|x, c; θ g ). ## analysis to extensively assess the effectiveness of the proposed methods, we conduct the following in-depth analyses. ## conclusions