Style versus Content: A distinction without a (learnable) difference? (2020)

Abstract: Textual style transfer involves modifying the style of a text while preserving its content. This assumes that it is possible to separate style from content. This paper investigates whether this separation is possible. We use sentiment transfer as our case study for style transfer analysis. Our experimental methodology frames style transfer as a multi-objective problem, balancing style shift with content preservation and fluency. Due to the lack of parallel data for style transfer we employ a variety of adversarial encoder-decoder networks in our experiments. Also, we use of a probing methodology to analyse how these models encode style-related features in their latent spaces. The results of our experiments which are further confirmed by a human evaluation reveal the inherent trade-off between the multiple style transfer objectives which indicates that style cannot be usefully separated from content within these style-transfer systems.

## introduction style transfer can be defined as a natural language generation (nlg) task where an input word sequence (a text, a sentence, an utterance. . . ) from an original style is rephrased in order to fit a target style while keeping its semantic information, leading to a style-transferred sequence. this task can be applied to many facets of the natural language: sentiment, complexity, authorship, formality, and so on. the primary use-case of textual style transfer is to improve language understanding between humans or between humans and machines by adapting messages to the social context or abilities of an interlocutor. however, work on style transfer can also provide insight into language more generally, for example by providing insight into the relationship between style and content. indeed, a pre-requisite of style transfer is the presumed ability to separate style from content, and so research on style transfer can inform whether computational models can usefully learn this distinction. research on style-transfer often focuses on one specific aspect of textual productions, (e.g., preserving the meaning of the original sequence) and disregards the others (e.g., how present is the target style in the generated output, and the fluency of the generated sentences). however, this paper highlights the importance of evaluating style transfer systems in terms of multiple objectives where fitting the target style is counter-balanced by the needs to generate a linguistically fluent output sequence and at the same time preserving the original meaning. furthermore, based on an analysis of a number of evaluation experiments of style-transfer systems, including a probing experiment that analyses the latent space of these systems, we identify an inherent trade-off in style-transfer between transfer strength, fluency and content preservation. we conclude that style and content cannot be usefully disentangled by these systems (at least not for the task of sentiment transfer), and so a holistic perspective on style and content that focuses on finding a suitable trade-off for the target application is likely the best approach to developing a useful style-transfer system. the paper structure is as follows. section 2 reviews related work on style-transfer and distinguishes between: (i) systems that conceive of style as being encoded in a set of explicitly identifiable features that can be removed and replaced, and (ii) systems that do not explicitly identify stylistic features. section 3 introduces the style transfer evaluation objectives we use; section 4 describes the models we evaluate in our experiments, and section 5 describes the dataset and our experimental methodology. in section 6 we evaluate the style-transfer models, and in section 7 probe their representations of the input to examine what stylistic information they encode. in section 8 we set out our conclusions and future work. ## literature review to deal with textual style transfer, a first group of approaches consider a two-step process where markers of the source style are removed before generating the target style output sequence. marking and removing the original style features can be achieved through frequency-based methods @xcite or neural networks that integrate attention modules @xcite . the generation step relies on techniques such as the retrieval of textual segments from a corpus of the target style @xcite , or employing neural network generation techniques @xcite . the second group of approaches which have been mostly implemented in the literature focus on endto-end learning strategies to frame the task of style transfer. end-to-end learning approaches enable learning of a latent representations of an input @xcite , encoding potentially both the input content and style. historically, such style transfer approaches have been inspired by the neural machine translation techniques, basically a sequence-to-sequence encoder-decoder architecture based on recurrent neural networks @xcite . @xcite applied this approach to text simplification and summarization (considering the style here as "verbosity"). the key idea is that the text embedding vector generated by the encoder is a style-free representation of the input text @xcite . a major limitation of this approach is the need for parallel data. in some scenarios this data challenge can be addressed by either relying on intermediate resources-e.g., using "zero-shot translation" @xcite -or monolingual data-e.g. using "back translation" @xcite . alternatively, generative adversarial networks @xcite have been implemented for style transfer, removing the need for parallel data. these models include a generator block which generates two sequences for each input sequence (a style transferred sequence and a reconstructed sequence), and a discriminator block (a classifier) which tries to guess whether a given sequence is style transferred or reconstructed. generators are usually standard encoder-decoders @xcite @xcite , but various refinements have been proposed. for instance, several generators (decoders) can be used, one for each style @xcite , or a variational encoder can be used for encoding @xcite . in most cases, the decoders are conditioned on a latent representation (embedding) of the input sequence concatenated with an embedding of the target output style. the concept of discriminator can also be extended, for instance in order to force encoded representations of the input to either focus on its style or meaning or both @xcite . ## evaluation aspects this section introduces the evaluation methodologies we use in our experiments. each of these methods is taken from the literature and each focuses on a different aspect of textual style transfer: content preservation, style transfer strength and fluency. style shift power style shift power investigates how well the output of a model fits with the target style. one approach is to employ a pre-trained classifier to measure the percentage of the style-shifted sequences which are labeled with the desired style @xcite @xcite @xcite @xcite . here, we employ the same classifier type as @xcite , the textcnn model @xcite and train it on the yelp dataset. it achieves an accuracy of %97 on predicting sentiment (positive and negative classes) on the test set of the yelp dataset. content preservation many approaches, usually inherited from historical nlg tasks, have been employed to compare how well the generated output preserves the meaning of the original sequence. we compute a content preservation rate between an input sequence and a generated sequence as the cosine similarity between their embedded representations. the process we use to generated the embedding of a sequence follows the method proposed by @xcite . given a sequence of words @xmath0. . . w n ], each token x i is represented by an embedding e @xmath1. these embeddings are generated using a pre-trained 100-dimensional glove model @xcite . then, we calculate the min, mean and max pooling of these word embeddings, namely m, µ, m as follows: the embedding of the sequence x is then created by the concatenation of m, µ and m . in practice, sentiment markers foot_0 are removed from the sequence x to make sure the content preservation metric indeed measures the content similarity. finally, when processing a dataset, content preservation is averaged over all its sentences. fluency following @xcite and @xcite , we measure the fluency of an output sequences using language model perplexity. for our experiments, we trained an rnn-based language model, consisting of single-layer rnn with gru cells @xcite , on the yelp dataset. again, tokens are embedded using a 100-dimensional layer which was initialized at training time with a pretrained glove model @xcite . ## models this section introduces three models employed in this work to deal with textual style transfer: a stateof-the-art adversarial network, which we refer to as the base model (section 4.1); an encoder extension of the base model employing a novel type of encoder (in section 4.2); a generator extension of the base model employing multi-generators (section 4.3). ## multi-dimensional system evaluation this section reports on four multi-dimensional experiments on sentiment transfer. firstly, a comparison between the standard and elmo-based encoders is presented (section 6.1). then, attention shifts to the generator component (sections 6. 2 and 6.3 ). section 6.2 evaluates the benefits of using multiple style specific generators, and section 6.3 evaluates the effects on content preservation of reinforcing the latent representation of the input throughout the output generation process. finally, section 6.4 validates our multi-dimensional experimental framework for style-transfer against human evaluations. in the following sections, we will discuss the results of each of these experiments which are listed in ## probing the relationship between style and content in section 6.3 we observed that reinforcing the latent representation of the input (z) during generation could result in improved content preservation but also consistently resulted in a deterioration in styleshift power. in this section we examine whether this trade-off is a result of stylistic information being encoded within the latent representation of the input. inspired by the probing experiments described by @xcite , we designed a probing experiment to examine what style information each of our models encoded in their latent representations. to do the probing classification experiment, for each style-transfer framework we trained a classifier that took the z representation generated by the encoder of the framework and predicted the style of the input sequence that had been feed into the encoder. the idea here is that if we can train a classifier to accurately predict the style of an input from the z representation of an input sequence generated by the encoder, then this z vector must have information related to the style of the input sequence. therefore, by examining the accuracy of the style classifier for each style-transfer framework we can get an insight into the amount of stylistic information the framework encodes in its z space. for each style-transfer framework, we trained a separate feed-forward network with a single hidden layer and a sigmoid output layer as the classifier using pairs of z vectors (embedded representations) of the input sequences and their original style. we employed the yelp review dataset for training where the distribution of the "positive" and "negative" labels are balanced. ## further discussion and future steps our results indicate that style cannot be totally separated from content. this is because encoders mistakenly strip out content when attempting to remove source stylistic features. this is most clearly seen in the interaction between the results of the probing experiment and the content preservation results for the different systems. the probing experiment revealed that reduction in encoding the stylistic features in the latent representation of the inputs as a result of reinforcing the content ( to summarize, the results of this study show the necessity of employing style transfer-specific encoders as opposed to pre-trained embeddings. furthermore, the results indicate that each of the tested frameworks has its own strength and weaknesses. indeed, our result indicate the benefits of framing style-transfer as a multi-dimensional task: given the trade-off we observed in terms of style-shift and fluency versus content preservation it is important to consider all of these aspects when evaluating a system as understanding the trade-off systems make can inform the design of improved style-transfer systems. finally, our results suggest that stylistic and contextual features cannot really be separated or at least not in the case context of framing sentiment as a style. overall, we consider the multi-generator framework as a good basis for future research, and we plan to implement alternative techniques to improve content preservation power in this architecture. furthermore, considering the significance of doing the evaluation in a comprehensive manner, we would like to focus on introducing a single evaluation metric for this task which takes the three evaluation aspects of style-shift, content maintenance and fluency into consideration.