Thank you BART ! Rewarding Pre-Trained Models Improves Formality Style Transfer (2021)

Abstract: Scarcity of parallel data causes formality style transfer models to have scarce success in preserving content. We show that fine-tuning pre-trained language (GPT-2) and sequence-to-sequence (BART) models boosts content preservation, and that this is possible even with limited amounts of parallel data. Augmenting these models with rewards that target style and content –the two core aspects of the task– we achieve a new state-of-the-art.

## introduction and background style transfer is the task of automatically converting a text of one style into another, such as turning the formal "i viewed it and i believe it is a quality program." into the informal "i've watched it and it is awesome!!!!". this task, which can be used for, e.g., personalised response generation, translation of ancient text into modern text, and text simplification, is particularly challenging since style must be changed while ensuring that content is preserved. accordingly, the performance of style transfer systems is commonly assessed on both style strength and content preservation. due to the general scarcity of parallel data, unsupervised approaches are popular. these include disentangling style and content by learning a distinct representation for each @xcite @xcite , and back translation @xcite @xcite . a common strategy to enhance style accuracy is to introduce a reward in the form of a style classifier @xcite @xcite @xcite . as a result, unsupervised models achieve good accuracy in style strength. content preservation is however usually unsuccessful @xcite . parallel data can help to preserve content, but is limited. @xcite combine the train sets of two different domains and incorporate machine translation to train their models with a multi-task learning schema, plus model ensembles. @xcite use it to train a supervised sequence-tosequence model, and in addition to the commonly used style strength reward, they include a reward based on bleu @xcite to enhance content preservation. @xcite propose a semi-supervised model combining parallel data with large amounts of non-parallel data. pre-trained models, successful in a variety of nlp tasks, have recently been used in formality style transfer. @xcite propose several data augmentation methods for pre-training a transformer-based @xcite model and then used gold data for fine-tuning. using gpt-2 @xcite , @xcite and @xcite propose a harness-rulebased preprocessing method, and joint training of bi-directional transfer and auto-encoding with two auxiliary losses. contemporary work by @xcite develops a semi-supervised model based on bart large @xcite . contributions focusing specifically on formality transfer, for which parallel data is available, (i) we take the contribution of pre-trained models a step further by augmenting them with reward strategies that target content and style, thereby achieving new state-of-the-art results. (ii) we analyse separately the contribution of pre-trained models on content and style, showing that they take care of preserving content (the hardest part of style transfer to date), while ensuring style strength. (iii) moreover, experimenting with training size, we show that while parallel data contributes to content preservation, fine-tuning pre-trained models with 10% of parallel data is more successful than training on 100% of data from scratch. reducing the need for parallel data opens up the applicability of gpt-2 autoregressive decoder [bos] [src] [sep] [tgt] [eos] reward [bos] [src] [sep] [tgt] [eos] bleu score / style classifier (a) architecture of the gpt-2-based model bart [bos] [src] [eos] [bos] [tgt] [eos] bidirectional encoder autoregressive @xmath0: model architectures. we use three special symbols: [bos] in front of every source sentence, [sep] between the source and target sentences (only in gpt-2), and [eos] at the end of every target sentence. ## method we propose a framework to control the style of output text for style transfer atop pre-trained models. given a source sentence @xmath1, • • • , x n } of length n with style s 1 and a target style sentence @xmath2, • • • , y m } of length m with style s 2 , our model aims to learn two conditional distributions, altering the style of a sentence while preserving its original content. our framework consists of (i) fine-tuning pre-trained models on a formality transfer parallel corpus; (ii) incorporating rewards to enhance style change and content preservation. ## conclusions fine-tuning pre-trained models proves a successful strategy for formality style transfer, especially towards content preservation, thereby reducing the need for parallel data. a sequence-to-sequence pre-trained model (bart) outperforms a language model (gpt-2) in content preservation, and overall, and with the addition of rewards achieves new stateof-the-art results. the fact that gpt-2 is instead often better at style strength could be (partly) due to how the style reward is implemented in the two models (eq. 4 and 5), and will need further investigation. for a better understanding of the different behaviour of bart and gpt-2 for this task, the next natural step is to include human evaluation.