S tyle PTB : A Compositional Benchmark for Fine-grained Controllable Text Style Transfer (2021)

Abstract: Text style transfer aims to controllably generate text with targeted stylistic changes while maintaining core meaning from the source sentence constant. Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes (e.g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence. In this paper, we introduce a large-scale benchmark, StylePTB, with (1) paired sentences undergoing 21 fine-grained stylistic changes spanning atomic lexical, syntactic, semantic, and thematic transfers of text, as well as (2) compositions of multiple transfers which allow modeling of fine-grained stylistic changes as building blocks for more complex, high-level transfers. By benchmarking existing methods on StylePTB, we find that they struggle to model fine-grained changes and have an even more difficult time composing multiple styles. As a result, StylePTB brings novel challenges that we hope will encourage future research in controllable text style transfer, compositional models, and learning disentangled representations. Solving these challenges would present important steps towards controllable text generation.

## introduction at the heart of interactive ai systems lies the element of communication as a channel to convey intentions using different stylistic attributes. research in human-ai interaction has focused on building dialog systems @xcite , virtual assistants @xcite , and intelligent agents @xcite @xcite that can communicate their intentions with specific styles for different situations, target audiences, and environments @xcite the good service of the waitresses makes me dread going sometimes. the good service of the waitresses makes me enjoy going sometimes. i left three messages without a call back. i left three messages. i left three thankful messages. after 3 months they can't be too new now. after 3 months they can't be too new. ## related work several lines of research have aimed to formalize styles in natural languages through computational and linguistic perspectives @xcite . the first systematic formulation of styles was by @xcite and later extended by @xcite to 4 representational categories including lexical, syntax, thematic, and semantic aspects. following this, there has been some early efforts applying stylistic analysis into dialog generation @xcite , machine translation @xcite , and text generation @xcite . we take advantage of this prior work when formalizing our new styleptb dataset. current benchmarks for style transfer focus on high-level style definitions such as transfer of sentiment @xcite @xcite , politeness @xcite , formality @xcite @xcite , writing styles @xcite @xcite and some other styles @xcite . however, these only focus on only high-level styles, unlike styleptb. computational models for style transfer span statistical nlp methods @xcite , neural generative models @xcite @xcite , and retrieve-and-edit approaches @xcite @xcite @xcite . these approaches work for a predefined set of styles but are unable to generalize to compositions of styles. evaluating style transfer is difficult due to the diversity of plausible transferred sentences. in addition to automatic scores such as bleu, perplexity, or binary classification accuracy of style transfer @xcite @xcite , other automatic metrics @xcite and human evaluation are also commonly used @xcite . ## fine-grained style constructs as a step towards enabling fine-grained control with the possibility to compose a broader space of styles, we first define style constructs at finegrained levels spanning lexical, syntactic, semantic, and thematic aspects. when selecting these style constructs, we have 2 goals in mind: (1) they should be representative of the four aspects (lexical, syntactic, semantic, thematic) following the formal categorizations in @xcite , and (2) the transfers should be consistent (i.e. welldefined such that if multiple annotators are asked to modify the same sentence, the results will be similar). with these goals in mind, we summarize the ## the styleptb dataset using these selected 21 style constructs, we now illustrate the steps towards collecting and annotating parallel sentences across style transfers. ## a model for compositional transfer we extend the pre-trained gpt2 language model @xcite for parallel style transfer by giving it designated style transfer tokens as input in addition to the source sentence. for example, for each individual binary style s i , we define a style transfer token ∆ @xmath0, 1, 2} where ∆ @xmath1, ∆ @xmath2, and vice versa for ∆ @xmath3. we likewise extend the definition of ∆ i for styles taking more than 2 values. given a parallel (source, target) pair (s, t), we define the appropriate transfer token ∆ ∈ {0, 1, 2} as a step towards compositionality, we also train with (source, target) pairs that undergo multiple atomic style transfers as provided in styleptb, resulting in multiple style transfer tokens ∆ i being activated at the same time. we call the resulting model cs-gpt (compositional style gpt) and show its architecture in figure 4 . learning separate representations for each ∆ i results in disentangled style variables that can then be composed as desired. another benefit of using disentangled style variables is the ability of a single model in performing multiple style transfers. ## experiments we test the performance of current style transfer models on styleptb. anonymized data and code is included in the supplementary, and we present extra details and results in appendix b and c. ## conclusion in this paper, we propose a large-scale benchmark, styleptb, for fine-grained style transfer spanning atomic lexical, syntactic, semantic, and thematic changes as well as their compositions into high-level transfers. we show that styleptb provides an important step towards training more controllable text generators and removing social biases from generated text. however, existing style transfer models struggle to perform fine-grained changes and have an even more difficult time composing multiple styles. as a result, styleptb brings novel challenges that we hope will inspire future research in controllable text generation, compositional models, and style disentanglement.