Challenges in Emotion Style Transfer: An Exploration with a Lexical Substitution Pipeline (2020)

Abstract: We propose the task of emotion style transfer, which is particularly challenging, as emotions (here: anger, disgust, fear, joy, sadness, surprise) are on the fence between content and style. To understand the particular difficulties of this task, we design a transparent emotion style transfer pipeline based on three steps: (1) select the words that are promising to be substituted to change the emotion (with a brute-force approach and selection based on the attention mechanism of an emotion classifier), (2) find sets of words as candidates for substituting the words (based on lexical and distributional semantics), and (3) select the most promising combination of substitutions with an objective function which consists of components for content (based on BERT sentence embeddings), emotion (based on an emotion classifier), and fluency (based on a neural language model). This comparably straight-forward setup enables us to explore the task and understand in what cases lexical substitution can vary the emotional load of texts, how changes in content and style interact and if they are at odds. We further evaluate our pipeline quantitatively in an automated and an annotation study based on Tweets and find, indeed, that simultaneous adjustments of content and emotion are conflicting objectives: as we show in a qualitative analysis motivated by Scherer’s emotion component model, this is particularly the case for implicit emotion expressions based on cognitive appraisal or descriptions of bodily reactions.

## introduction humans are capable of saying the same thing in many ways. careful lexical choices can re-shape a concept in different modes of presentation, giving it a humourous tone, for example, or some degree of formality, or a rap vibe. this type of linguistic creativity has recently been mirrored in the task of textual style transfer, where a stylistic variation is induced on an existing piece of text. the core idea is that texts have a content and a style, and that it is possible to keep the one while changing the other. past work on style transfer has targeted attributes (or styles) like sentiment @xcite and tense @xcite , producing a rich literature on deep generative models that disentangle the content and the style of an input text, and subsequently condition generation towards a desired style @xcite @xcite . with this paper, we propose a non-binary style transfer setting, namely emotion style transfer, in which the target corresponds to one emotion (following ekman's fundamental emotions of anger, fear, joy, surprise, sadness, and disgust). further, this setting is particularly challenging as emotions are on the fence between content and style. to the best of our knowledge, this type of attribute has been explored only to some degree by the unpublished work by @xcite , who transfer text towards 20 affect-related styles. emotions received more attention in conditioned text generation @xcite @xcite . to explore the challenges of emotion style transfer (for which we depict an example in figure 1 ), we develop a transparent pipeline based on lexical substitution (in contrast to a black-box neural encoder/decoder approach), in which we first (1) select those words that are promising to be changed to adapt the target style, (2) find candidates that may substitute these words, (3) select the best combination regarding content similarity to original input, target style, and fluency. as we will see, this straight-forward approach is promising while it still enables to understand the changes to the text and their function. emotions are not only interesting from the point of view that they contribute to content and style. they are also a comparably well-investigated phenomenon with a rich literature in psychology. for instance, @xcite states that emotions consist of different components, namely a cognitive appraisal, bodily symptoms, a subjective feeling, expression, and action tendencies. descriptions of all these components can be realized in natural language to communicate a specific private emotional state. we argue (and analyze based on examples later) that a report of a feeling ("i am happy") might be challenging in a different way than descriptions of bodily reactions ("i am sweating") or events ("my dog was overrun by a car"). with our white-box approach of style transfer and the evaluation on the novel task of emotion transfer, we address the following research questions: to what extent can lexical substitution modulate the emotional leaning of text? what is its limitation (e.g., by changing the emotion "style", does content change as well)? our results show that the success of this approach, both in terms of style change and content preservation, depends on the strategies used for selection and substitution, and that emotion transfer is a viable task to address. further, we see in a qualitative analysis that what an emotion classification model bases its decisions on might not be sufficient to guide a style transfer method. this becomes evident when we compare how transfer is realized across types of emotion expressions, corresponding to specific components of scherer's model. our implementation is available at http://www. ims.uni-stuttgart.de/data/lexicalemotiontransfer . ## methods emotion transfer can be seen as a task in which a sentence s is paraphrased, and the result of this operation exhibits a different emotion than s, specifically, a target emotion. we address emotion transfer with a pipeline in which each unit contributes to the creation of emotionally loaded paraphrases. the pipeline is shown in figure 2 . first is a selection component, which identifies the tokens in s that are to be changed. then, the substitution component takes care of the actual substitution. it is responsible for finding candidate substitutes for the tokens that have been selected, producing paraphrases of the input sentence. importantly, paraphrases are over-generated: at this stage of the pipeline, the output is likely to include sentences that do not express the target emotion. paraphrases are then scored and re-ranked in the last, objective component, which picks up the "best" output. ## experiments having established the general pipeline, we move on to the question whether our strategies for selection and substitution actually produce variations with the desired emotion (rq1). in addition, we examine the interaction between the emotion connotation of the paraphrases and their similarity to the inputs (rq2). these questions are answered in an automatic and a human evaluation. ## analysis we now turn to a more qualitative analysis of the results. due to space restrictions, we show examples for the four pipeline configurations, all with the same objective function emo(•)+sim(•)+flu(•) and a comparison of the at+in model with different objective functions in supplementary material upon acceptance of this paper. here, in figure 2 , we focus on a discussion of those cases which we consider particularly difficult, though common in everyday communication of emotions. in the selection of these examples, we follow the emotion component model of @xcite and use two examples, which correspond to a direct (explicit) communication of a subjective feeling (ex, id 1, 2), the description of a bodily reaction (br, id 3, 4), and a description of an event for which an emotion is developed based on a cognitive appraisal @xcite @xcite . the examples which communicate an emotion directly are challenging because there is no other content available than the emotion that is described (id 1, 2). the model has the choice to exchange two out of three words, and in nearly all cases, it choses to keep "i" and replaces the verb and the emotion word. while the latter is replaced appropriately, the verb is in most cases not substituted in a grammatically correct way. we see here that the emotion classification component in the objective function outrules the language model. this illustrates one fundamental issue with presumably all existing affect-related style transfer method: the original emotion is turned into the target emotion, but their intensities do not correspond. in the examples which describe a bodily reaction (id 3, 4), we see that the attention mechanism does not allow the words "over my face" or "trembling" to change. instead, it finds the other words more likely to be substituted -the classifier is not informed about the meaning of "trembling" and "over my face". the substituted words make sense, but content and fluency are sacrificed again for the maximal emotion intensity available. similarly, the emotion classifier and therefore the associated attention mechanism do not find "close to the street" to be relevant to develop an emotion (id 5). instead, other words are exchanged to introduce the target emotion. these issues are mostly due to issues in the emotion classification module. further, we see that the substitution and selection elements might have a higher chance to perform well if they considered phrases instead of isolated words. we observe a lack of fluency in many of our output sentences, which we attribute to a dominance of the emotion classifier score. adapting the weights of the scores in the objective might have potential, however, our findings might suggest that content, emotion and fluency are in conflict with each other -and that obtaining a particular emotion is only possible by sacrificing content similarity. not doing so seems to lead to non-realistic utterances. ## conclusion & future work with this paper, we introduced the task of emotion style transfer, which we have seen to be particularly difficult, on the one side due to being on the fence between content and style, and on the other side due to being a non-binary problem. our quantitative analyses have shown that there is indeed a trade-of between content preservation and obtaining a target style and that emotion transfer is especially challenging when the text consists of descriptions of emotions in which the separation between content and style is not linguistically clear (as in "i am happy that x happened"). we propose that such test sentences based on descriptions of bodily reactions and event appraisal will be part of future test suits for emotion style transfer, in order to ensure that this task does not work well only on particular expressions of emotions. we identified the challenge to find the right tradeof between fluency, target emotion, and content preservation. this is particularly challenging, as it would be desirable to separate the emotion intensity from our objective function. we therefore propose that intensity is handled as a fourth component in future work. this could be combined with a decoder as suggested by @xcite . finally, a larger-scale human evaluation should be carried out to clarify the contribution of each component.