Lost in Back-Translation: Emotion Preservation in Neural Machine Translation (2020)

Abstract: Machine translation provides powerful methods to convert text between languages, and is therefore a technology enabling a multilingual world. An important part of communication, however, takes place at the non-propositional level (e.g., politeness, formality, emotions), and it is far from clear whether current MT methods properly translate this information. This paper investigates the specific hypothesis that the non-propositional level of emotions is at least partially lost in MT. We carry out a number of experiments in a back-translation setup and establish that (1) emotions are indeed partially lost during translation; (2) this tendency can be reversed almost completely with a simple re-ranking approach informed by an emotion classifier, taking advantage of diversity in the n-best list; (3) the re-ranking approach can also be applied to change emotions, obtaining a model for emotion style transfer. An in-depth qualitative analysis reveals that there are recurring linguistic changes through which emotions are toned down or amplified, such as change of modality.

## introduction the quality of machine translation (mt) models in some areas follows close behind that of humans @xcite . mt is deployed widely to support human-to-human communication across languages, e.g., in chat systems, customer support, or (social) media. it is also employed in downstream nlp tasks such as sentence simplification @xcite , error correction @xcite , paraphrasing @xcite , or cross-lingual resource creation @xcite . with the increasing use of mt, however, expectations about output quality also grow, and now that the goal of adequacy with regard to propositional content is met more often than not, more subtle aspects start receiving attention. one such aspect is affective content. establishing common ground is essential for successful mt-assisted communication @xcite , but it is still unclear how well mt promotes this, especially when handling the affective qualities of texts. on the one hand, it is able to mostly preserve author sentiment @xcite . on the other, it is known that translation obfuscates some socio-demographic characteristics of authors, like gender and personality traits @xcite . in this paper, we investigate the question of how well emotions are preserved in mt. answering this question and, if necessary, increasing the degree of emotion preservation, is important both theoretically (to inform cross-lingual studies that use translation as part of their experimental setup) and practically (to improve the usefulness of mt). the starting point of our research is a study by @xcite , who show that some semantic nuances tend to vanish in translation. in fact, just like human translation, mt is not guaranteed to preserve any of the linguistic properties of input texts (e.g., politeness markers may exist in one language but not in the other, passive sentences may be turned into active, metaphoric expressions can be rendered by a more literal paraphrase). therefore, to counteract the fading of emotions through translation, we establish emotion-based translation candidate re-ranking that is research question sentence (input) he was furious at the apparent disregard for rules. ## a method for emotion preservation in neural machine translation we conceptualize emotion preservation in nmt as a post-processing re-ranking step. as shown in figure 1 , this involves three components: a translation model, an emotion classifier, and a candidate selection procedure. starting from an input in source language s, we generate the n-best translation candidates in a target language t with an nmt system, which is presumably agnostic to emotion-specific considerations. then, we re-rank these candidates based on probabilities produced by an emotion classifier, and select the best hypothesis given those emotion-level considerations. hence, the crucial variable is the diversity of the n-best list: the more diverse, the better the emotion classifier can promote hypotheses that express particular emotions even if they are not optimal from the point of view of the overall scoring function of the nmt system. translation model. we require a translation model that returns a list of n-best translation candidates, which is the case for essentially every statistical or neural mt system. we use fairseq @xcite , an open-source sequence-to-sequence modeling toolkit applicable to various tasks, mt included. it shows state-of-the-art performance and it was developed with the goal to replicate other model architectures. therefore, we assume that it is reasonably representative for other models. importantly, fairseq supports different search algorithms, like beamsearch and top-k sampling, which differ in their ability to encourage diversity in the output. beamsearch searches the space of hypotheses left-to-right, retaining at each time step a number of top-scoring candidates that equals the width of the beam, and expanding on those. sequences decoded with beamsearch differ on minimal portions @xcite , while they are more varied when generated with sampling strategies. top-k sampling, for instance, does not aim at maximizing the likelihood of text. instead, it randomly samples words step-wise and outputs from the top-k most probable ones @xcite . ## experimental setup: back-translation the most natural setup to study emotion preservation in translation, and the framework outlined in the previous section, would be bilingual: analyzing the translation of some source language text into a target language. for instance, one could compare the distribution of emotion probabilities for a translation against the corresponding distribution for the source text. however, a meaningful cross-lingual comparison of emotion probabilities is methodologically challenging: this would require either manual annotation or highly comparable emotion classifiers for several languages. manual annotations are costly, and emotion annotation is known to be tricky in terms of intersubjective replicability @xcite . neither are we aware of emotion classifiers with evidently similar behavior across languages. to circumvent this problem, our experimental setup uses a back-translation version of the method described above, shown in figure 2 . we compose two translation steps (@xmath0. formally, given an input in s and a target emotion, we generate the best translation in t; this is then translated back into multiple hypotheses, providing a set of n paraphrases for the original input. we acknowledge that this type of setup is a conceptual simplification of the problem, which does not measure the loss of emotion in one direction, nor accounts for where the change in emotion occurs. as a matter of facts, it risks overestimating the absolute magnitude of problems in emotion preservation in mt, but this is a price to pay for the usage of our monolingual emotion classifiers. on the other hand, we can still compare the magnitude of emotion loss across different mt settings (which we do in the sections below). in addition, results that would indicate that we can improve emotion preservation in back-translation would conversely be stronger than such results obtained on a single translation step. ## conclusion and future work our goal was to understand if automatic translation retains the emotional substance of texts. we found that a state-of-the-art nmt system tends to tone down emotion connotations, thus presenting a problem rq emotion sentence a a. when i have to take exams i am very excited and have not much time for the housekeeping. then my friend has to do everything 1 g when i have to take exams, i am very excited and do not have much time for the budget. then my boyfriend has to do everything. 2 a when i have to take exams, i am very excited and have little time for housekeeping. then my girlfriend has to do everything she can. 3 j when i have exams, i'm very excited and i don't have much time for the household. then my boyfriend has to take care of everything. a b. when a friend told me a story and i stayed dumb because i had no story to tell. 1 f when a friend told me a story and i remained silent because i had no story to tell. 2 a if a friend told me a story and i was mute because i had no story to tell. 3 g when a friend told me a story, i stayed silent because i had nothing to tell. d c. on new years eve i drank too much alcohol, so much that i had to vomit in the presence of other people. 1 sh on new year's eve i drank so much alcohol that i vomited in the presence of other people. 2 d on new year's eve i drank so much alcohol that i had to vomit in the presence of other people. 3 g on new year's eve, i drank too much alcohol, so much that i threw up in the presence of other people. d d. i feel disgusted with the bootlickers, with helpless people. 1 sa i loathe the bootleggers, the helpless people. 2 d i am disgusted by the boot-lickers, by the helpless people. 3 sh i loathe boots, i loathe helpless people. j e. when a person that i like very much got near to me. 1 f if a person i like very much approached me. 2 j if a person i like very much got close to me. 3 d if someone i like came up to me. f f. i was going to knock down a pedestrian with my car. 1 a i was trying to push a pedestrian over with my car. 2 f i was going to knock over a pedestrian with my car. 3 j i wanted to overturn a pedestrian with my car. sh i tried to knock over a female pedestrian with my car. for the development of affect-aware mt products, for cross-lingual research based on the translation of existing data, and for communication aided by mt. we showed how an emotion-informed subselection of translation candidates can improve this situation without adversely affecting translation accuracy. moreover, we used the same post-processing methodology to induce emotion variability and address the task of emotion style transfer. results show that mt outputs can be improved in their emotion rendering, but we relied on a backtranslation pipeline instead of a real-world translation scenario. this motivates an important next research step, namely the development of an emotion classifier which estimates emotion probability distributions in multiple languages in a comparable manner. it is still open how to measure comparability and how to optimize that measure. finally, our analysis relied on a single nmt system, namely fairseq. despite our argument that this tool is representative for a range of systems, our study should be extended to other systems and other target languages.