Unsupervised Text Style Transfer with Content Embeddings (2021)

Abstract: The style transfer task (here style is used in a broad “authorial” sense with many aspects including register, sentence structure, and vocabulary choice) takes text input and rewrites it in a specified target style preserving the meaning, but altering the style of the source text to match that of the target. Much of the existing research on this task depends on the use of parallel datasets. In this work we employ recent results in unsupervised cross-lingual language modeling (XLM) and machine translation to effect style transfer while treating the input data as unaligned. First, we show that adding “content embeddings” to the XLM which capture human-specified groupings of subject matter can improve performance over the baseline model. Evaluation of style transfer has often relied on metrics designed for machine translation which have received criticism of their suitability for this task. As a second contribution, we propose the use of a suite of classical stylometrics as a useful complement for evaluation. We select a few such measures and include these in the analysis of our results.

## introduction in this paper we consider the problem of unsupervised holistic textual style transfer -both the accomplishment of the task, as well as its evaluation. the "style" of text is roughly the way in which a text communicates its content. it might be thought of as the "voice" characteristic of a given author, an emergent quality that encompasses a wide range of (more or less measurable) characteristics such as register, sentence structure, and vocabulary choice. holistic style transfer takes a given text -written a priori in one "style" -and then rewrites it (preserving its content) in another style. holistic style transfer is distinct from more narrow style modification techniques which manipulate specific characteristics of prose such as formality, simplicity, or sentiment. to illustrate our idea of holistic style consider the following pair of translations of the opening lines of the aeneid of virgil. arms and the man i sing, the first who came, compelled by fate, an exile out of troy... @xcite i sing of arms and the man who of old from the coasts of troy came, an exile of fate, to italy and the shore of lavinium... @xcite as another example compare verses from two different "versions" of a fixed verse from the book of genesis, the first from the king james version and a river went out of eden to water the garden; and from thence it was parted, and became into four heads. and the second, the same verse, but in the new international version: a river watering the garden flowed from eden; from there it was separated into four headwaters. in both example pairs we can see that the content in the passages is the same, but the (holistic) style differs noticeably. the examples above are effectively examples of a human-executed style transfer. the potential applications of a machine holistic style transfer are numerous. for example, various periodicals often try to have a single "voice" and an unsupervised style transfer of the kind studied here would enable a staff writer to produce the content required of an article which was then "stylized" per the requirements of the venue. thus, a style transfer platform could be a high-powered editorial assistant. such a platform could also assist aspiring writers. all that said, one should not be blind to the more nefarious potential of successful style transfer machinery which could be useful for spoofing an audience to productive, or unproductive writerly ends (nature, 2020). one machine learning approach to holistic style transfer is to adopt and adapt the frameworks of translation models, treating each style as a language. along these lines, much of the existing research on this task depends on the use of parallel datasets, a schema that follows early work in machine translation, but parallel datasets in this domain are in fact rare. this motivates our approach wherein we continue to be inspired by machine translation work and employ recent results in unsupervised cross-lingual language modeling (xlm) to effect holistic style transfer while treating the input data as unaligned, an important next step in advancing this area in light of the scarcity of parallel datasets. additionally, we show that modifications to this framework which take advantage of the differences between the style transfer and machine translation tasks can improve model performance. specifically, we add "content embeddings" to the xlm which capture human-specified groupings of subject matter and observe improvement over the baseline model for a range of metrics. that brings us to the paired challenges resident in evaluting a style transfer technique. this task is complicated by the emergent nature of style. the analogy of style transfer to translation and concomitant efforts to use techniques from machine translation for the style transfer task have inspired the importation of evaluation metrics from machine translation to the style transfer setting, @xcite @xcite , although not without criticism @xcite . as the evaluation of a style transfer task should a priori measure the similarities of source and target texts to their "native environments", it seems natural to bring to bear some of the techniques from the field of stylometry, a discipline focused on the quantitative analysis of textual style. stylometry (or stylometrics) was born of a nineteenth century effort to settle -quan-titatively -scholarly dispute around the temporal ordering of plato's dialogues @xcite ). for this task, over 500 individual and measurable textual characteristics were identified. since that time stylometrics have been used (most famously) to address questions of disputed authorship (see e.g., @xcite ; @xcite ). if we imagine a system which perfectly performs style transfer as we have defined it, then the output of the system -in terms of its individual characteristics -should be stylistically indistinguishable from text written by the author whose style is targeted. it thus seems natural to use a range of stylometric measures used in the past to distinguish between authors' styles as an evaluation for the performance of such a system. this line of reasoning motivates a second contribution of this work wherein we introduce the idea of using stylometric measures for evaluation. we evaluate our systems using several stylometric measures in addition to the more commonly previously used metrics and show that the stylometric approach is a useful domain specific complement to translationbased metrics for the evaluation of the complex, subtle, and important task of style transfer. ## related work style transfer has some long roots. it is possible to frame the early work on text simplification (e.g., specia (2010)) or paraphrasing @xcite as a form of style transfer. style transfer research makes use of a range of datasets for training and evaluation. examples include the corpus of shakespearean plays and their "translations" into contemporary english @xcite for paraphrasing and a corpus of wikipedia pages and their simplified versions @xcite which is used for the general task of text simplification. the more stylistic features that are incorporated into building a model, the closer it gets to the kind of holistic effort we have described above. to that end, we highlight @xcite wherein a supervised style transfer model is developed which focuses on the modification of prose with respect to six aspects of style, including register, sentiment, focalisation, and prolixity. a broader approach for supervised holistic style transfer is addressed in @xcite ; @xcite . they make use of a model that depends on a corpus of versions of the bible, a priori aligned through the canonical and shared structuring of book, chapter, and verse, to learn the differences between examples written in different styles. unsupervised methods pose new challenges for style transfer. previous related work uses unsupervised training for generating text in a particular style. this includes the generation of stylized text @xcite and modification of the sentiment or formality of prose @xcite @xcite . there have also been advances in the use of unsupervised approaches for machine translation. many of these rely on the idea of back-translation @xcite to automatically generate a synthetic parallel from unaligned data. lample and conneau (2019) uses this concept along with a novel cross-lingual language model objective for pre-training to achieve impressive performance on the unsupervised translation task. ## conclusion and future work the task of holistic textual style transfer requires a system to take text in a native (source) style as input and then rewrite the text, retaining the meaning while changing the style consistent with a specified target. in many potential applications this task will need to be performed in contexts where there is no parallel data which captures the styles of interest available for training. examples range from the journalistic (writing articles in a given editorial style) to the literary (writing the style or voice of a given author). contexts such as these have large corpora of source and target examples, butpresumably -no source/target pairings. in this work we demonstrated that a modern unsupervised machine translation technique could be applied to unsupervised holistic textual style transfer in the context of different styles (well known and publicly available versions) of the bible. we the broader range of possible stylometric evaluation measures suggests that at least with respect to evaluation, a requirement of perfect evaluation and parallel texts might be relaxed. while the bible may seem to be particularly suited to the partition into content classes we employ, we believe this technique can be directly applied to many other textual sources as well. similar to bible versions, many translations exist of other classical works such as the epics written by homer or dante. in many of these translations alignment does not exist line by line so traditional supervised methods are not applicable. they are however "softly aligned" by book or chapter making content embeddings a natural choice. a model trained on these could then produce homer's iliad in the style of a translator who only produced a version of the odyssey. similarly, many translations of classic non-english novels exist and this system could be used to create a new translation targeting the style of a particular translator. demand for english-to-english style transfer also exists commercially. examples here include poetry parodies @xcite , continuations of stories from famous authors (james, 2011), or modernized retellings of stories @xcite . in these cases the content of the text either exists publicly or is written by the author. the style however is intentionally changed, either to match the works of another writer, or to remove the idiosyncrasies of the original style. unsuper-vised style transfer models could be used to help produce these works. in addition to these potential applications, our results reinforce the idea that consideration of content and style independently can improve the results of style transfer models. in cases where our technique cannot be directly applied, this provides additional evidence to researchers that finding a way to separate the two may improve results. in conclusion, this work highlights the utility of the bible as a dataset for holistic style transfer, demonstrates that unsupervised machine translation methods for holistic style transfer are possible and can be objectively evaluated, provides further evidence -and an actionable methodology -for the idea that learning content independent of style can be beneficial, and proposes the use of classical stylometric measures for evaluation of style transfer systems.