Analysing Mistranslation of Emotions in Multilingual Tweets by Online MT Tools (2023)

Abstract: It is common for websites that contain User-Generated Text (UGT) to provide an automatic translation option to reach out to their linguistically diverse users. In such scenarios, the process of translating the users’ emotions is entirely automatic with no human intervention, neither for post-editing, nor for accuracy checking. In this paper, we assess whether automatic translation tools can be a successful real-life utility in transferring emotion in multilingual tweets. Our analysis shows that the mistranslation of the source tweet can lead to critical errors where the emotion is either completely lost or flipped to an opposite sentiment. We identify linguistic phenomena specific to Twitter data which pose a challenge in translation of emotions and show how frequent these features are in different language pairs. We also show that commonly-used quality metrics can lend false confidence in the performance of online MT tools specifically when the source emotion is distorted in telegraphic messages such as tweets.

## introduction despite the tremendous improvement in the quality of automatic translation as a result of the use of neural machine translation (nmt) systems, nmt output still contains errors. this is particularly noticeable with user-generated text (ugt) such as tweets which do not follow the common lexico-grammatical standards @xcite . in spite of this limitation, nmt systems are commonly used in multilingual platforms such as twitter to provide its users with an idea of global views or emotions towards current events or public figures. in such scenarios, the component of the tweet that conveys emotions is often pivotal to the understanding of the tweet's message. there have been different studies which explored how far sentiment information can be captured from the machine-translated text @xcite @xcite . the objective of most research in this area, however, is from a sentiment classification perspective, rather than a translation accuracy perspective. it measures how far automatic translation of a language into english can help with the sentiment classification of that language by applying the available english sentiment resources on the target text. @xcite @xcite . the research presented in this paper, however, evaluates the preservation of the affect message from a user-related perspective. we assess how far nmt systems used in online platforms can be a successful real-life utility in transferring the user's fine-grained emotions such as anger and joy. research has shown that nmt models are capable of producing an impressively fluent output that completely misses the correct meaning of the source @xcite . the problem exacerbates when the source text is a deliberately concise text that carries a strong sentiment message as is the case with tweets. moreover, analysis of sentiment mistranslations produced by online tools revealed typical errors related to linguistic phenomena such as contronyms, idiomatic expressions, and dialectical code-switching @xcite . in this paper, we aim to investigate to what extent similar errors can be identified in the translation of tweets. to achieve this, we carry out an analysis of datasets of tweets automatically translated into different language pairs. at the end of this analysis, we attempt to provide answers to the following questions: to answer the above research questions, this paper is divided as follows: section 2 presents our data compilation process and the approach used for evaluating the translation of emotion in tweets by mt systems. section 3 analyses challenging features for the translation of emotions in multilingual tweets. it also provides a qualitative analysis of each feature based on its frequency in our compiled dataset and its prominence in each of the source languages explored. in section 4, we evaluate the efficacy of the mt automatic quality metrics in assessing the mistranlation of emotion within the multilingual ugt framework. section 5 briefly reviews relevant research which addressed the challenges in the automatic translation of sentiment. section 6 presents a conclusion on our experiment, limitations of the present study and recommendations for future research work. ## data collection and experiment setup in order to check how far automatic translation captures the specific emotion in tweets, we replicated a real-life scenario where mt systems are utilised spontaneously to translate the content of tweets. twitter currently supports built-in translations, so users can click on a translate tweet prompt visible directly under the tweet text to translate it. twitter mentions that it employs google translate for this service. to evaluate how far the mt system in this scenario can serve as a real-life tool, we used google translate api to automatically translate existing multilingual twitter datasets previously annotated for four emotions (joy, fear, aggression, and anger). it is important to note that these four emotions were chosen as representative of the common fine-grained sentiments expressed in tweets. the authors of tweets are usually either happy, angry, or fearful of something or someone, and their anger can either be aggressive or passive. the datasets were collated from different emotion-detection and aggression-detection shared tasks @xcite @xcite . the source datasets amounted to approximately 30,000 tweets in three languages: 23,000 in english, 4000 in arabic, and 3000 in spanish. we created two datasets from this source annotated data by using the google translate api. the first dataset was created by translating the arabic and spanish source datasets into english. the second dataset was created by translating part of the source english dataset into romanian, arabic, spanish and portuguese. these datasets were used to extract instances for our analysis. the next stage in our experiment was to extract instances in which the mt system may have failed to translate the emotion correctly. we call this failure "mistranslation of emotion" and it is identified by the discrepancy between the annotations of emotion in the source dataset and the emotions classified in the translated tweets. for example, if the original tweet is annotated as conveying 'anger' but a classifier predicted 'joy' for the translation, this pair was considered a potential mistranslation of emotion and was selected for manual analysis. to get the classifications of emotions in the translated tweets, we used the standard methodology employed in emotion classification. to this end, we built a classifier by fine-tuning a roberta xlm model @xcite on the previously annotated 23,000 source english tweets. this data was pre-processed by deletion of punctuation, non-alphanumeric symbols, lemmatisation, and lower-casing. we also used the demoji foot_0 python library to transfer the emojis into their equivalent lexicon (e.g. is translated into "dislike"). the english emotion-detection model was trained on four epochs and fine-tuned with the following adamw @xcite optimiser hyperparameters: learning @xmath0, β1 = 0.9, β2 = 0.98 and ϵ = 1 × e -8 . we divided the dataset into 90% training and 10% validation set. the validation accuracy reached 92%. the english classifier was used to predict the emotion of the google translate output for the translation of the arabic and spanish dataset into english. for the experiments where english was the source language, we classified the back translation of the english tweets. although the back translation may not be as accurate as the translated text, we opted for this compromise since ultimately the classifier's output will be manually compared to the source by human annotators. thus, the classifier's predicted emotion was compared to the gold-standard emotion of the source text, instances of discrepancy were extracted as potential mistranslations of emotions. ## analysis of challenging features to check the reasons for discrepancy between the predicted emotion and the emotion of the source text, a team of computational linguists who are native speakers of the analysed languages conducted a manual analysis on samples of the extracted potential mistranslations. the extracted samples for english to other languages amounted to 1600 tweets divided equally among the four target languages, and from the opposite direction, with english as a target language, they amounted to ≈3000 tweets divided between arabic and spanish as a source language. the disagreements in the dataset due to mistranslation of emotions are presented in figure 1 . spanish has clearly fewer cases of discrepant emotions in tweets, both when these are translated into english (≈8%) and when they are translations from english (≈27%). target languages like romanian and arabic show a much higher percentage of tweets with mistranslated emotions (61% and 41%, respectively). it is obvious from the analysed sample that some languages are more privileged than others in the real-life scenario we replicate for our experiment. next, we analysed in detail the linguistic features of instances where source tweet and translation have different emotion labels. the analysis showed that despite the unbalance in terms of mt accuracy among different language pairs as shown in figure 1 , there are common linguistic phenomena that cause distortion of emotion transfer among all the language pairs. based on our analysis of the sample dataset, we selected the six features that the annotators found to be commonly constituting a challenge in transferring emotions by the mt engine for all the studied language directions. these linguistic features are: hashtags, slang, non-standard orthography, idiomatic expressions, polysemy, ## measuring the transfer of sentiment from the analysis of these language features, it can be observed that using automatic translation tools for translating emotion in multilingual ugt such as tweets involves several linguistic challenges. from our manual analysis, we found that such challenges can lead to a severe distortion of the source emotive message. however, despite these challenges, nmt systems such as google translate are extensively utilised by social media platforms without human post-editing. in the research environment, the reliability of mt systems is commonly determined by automatic quality metrics that are domain agnostic as they evaluate the translation accuracy regardless of the type of source text. in this section we assess how far the commonly used quality metrics are able to signal out critical mistranslation of emotions as the ones analysed in the previous sections. the de facto standard for mt performance evaluation is the bleu score with its different variations @xcite . bleu gives equal penalty weight to inaccurate translation of n-grams, which may lead to performance overestimation (or underestimation). for example, the "joy" emotion in the tweet from our arabic dataset "what is this amount of happiness!" is flipped to anger by twitter's google translate tool which outputs "what is this amount of anger!". despite the distortion of the sentiment message, bleu only mildly penalises the swapping of the two opposite emotive nouns 'happiness' and 'anger' and this translation receives a bleu score of 0.76. the reason is that bleu gauges the performance of an mt model by an indiscriminate n-gram matching, regardless of the semantic weight of each word. by human standards, the mt performance in such cases is highly over-estimated. there have been numerous efforts to address the common pitfalls of n-gram-based metrics by incorporating semantic and contextual features in metrics specially when measuring the translation of sentiment @xcite . one very popular metric that has been introduced as a semantic-oriented metric is meteor @xcite . when it comes to evaluating an mt system performance in transferring emotion, even the semantically oriented automatic metrics do not give a penalty to a mistranslated sentiment proportional to the distortion it afflicts on the source message. for example, the negation in the arabic tweet "may god do not forgive those who put you in power" is missed in the mt output: "may god forgive the one who put you in power". the emotion is flipped from "anger" to "joy". despite the distortion of the emotion, the mistranslation receives a meteor score of 0.61. to quantify the ability of the bleu 3 and meteor metrics to assess the transfer of emotion in translated tweets, we selected an evaluation dataset consisting of 300 tweets extracted from the spanish and arabic dataset that was classified as having a mistranslation of emotion in the english translations. the tweets in this dataset were chosen in a way where the main error in the translation is the distortion of emotion due to one of the six linguistic features discussed in the previous sections. we also created another evaluation dataset consisting of 100 tweet/translation pairs with the same language directions where the online mt tool transferred the correct emotion. the evaluation datasets were translated by native speakers in the research team. the translators were also asked to assign a score to each pair of source-translation tweet, where 1 is the poorest sentiment transfer and 10 is best sentiment transfer. the average scores of annotators were taken as the final human score. we compared the human scores of the mistranslated tweets and the correctly translated tweets with bleu and meteor scores of their translations. we followed the wmt standard methods for evaluating quality metrics and used absolute pearson correlation coefficient r and the kendall correlation coefficient |τ | to evaluate each metric's performance against the human judgement. figures 2a and 2b show heatmaps visualising the pearson and kendall correlation coefficients for the mistranslated tweets, and figures 2c and 2d show the coefficients of the studied metrics with the correctly translated tweets. as seen from the figures 2a , and 2b , with the mistranslated tweets bleu score achieves only 0.22 and 0.17 pearson and kendall correlations, respectively. similarly, meteor records a pearson correlation of 0.26 but a relatively lower kendall correlation of 0.21. on the other hand, the correlation of the two metrics records (60%-68%) and (30%-60%) improvement on the correctly translated tweets for the pearson and kenall coefficients, respectively. our results show that conventional metrics' performance 3 we use the sacrebleu implementation of the bleu score for all the experiments @xcite . ## related research there has been a growing interest in analysing how far mt systems are capable of preserving the sentiment message, specifically in the automatic translation of tweets. salameh et al.(2015 ) acknowledge the fact that aspects of sentiment may be lost in translation, especially in automatic translation of arabic tweets. they show that the matching percentage between the manual sentiment annotation and an automatic sentiment annotation of the automatically translated dataset is 62.49% match as compared to the 68.65% match on a manually translated dataset. @xcite propose a method to reduce the mistranslation of sentiment in irish tweets. they manually expand the training data with an irish-language sentiment lexicon when building an irish-english mt system. the sentiment lexicon improves the sentiment accuracy of the translated text with an accuracy margin of 6%. @xcite argue that machine translation of ugt becomes more difficult because of the level of noise it contains. accordingly, the translation quality is affected in a way that may negatively impact sentiment preservation in the translation process. they show evidence of their analysis on a small dataset of 4000 english tweets and their translations in german. more recently, @xcite has shown that challenging features in tweets can lead to critical mistranslation of sentiment where the output of the mt system gives a deceivingly correct message that sometimes transfers a sentiment polarity opposite to the source tweet. as for the evaluation of the output of online mt tools, there have been several studies that address the shortcomings of conventional quality metrics such as bleu. for example, @xcite points to the inconsistencies of bleu as a parameterised metric since its score changes with a change of the parameters for tokenisation and normalisation scheme. @xcite demonstrate the inability of automatic metrics such as bleu and meteor to distinguish between a critical error that distorts the affect message in ugt data and a non-critical error where the mt affects the fluency of the source but still transfers the correct sentiment. ## conclusion in this research, we evaluated the ability of the mt online system to translate fine-grained emotions in tweets between different language pairs. our analysis has shown that there are linguistic features that are common among different language pairs which cause problems in translating tweets by nmt tools. more crucially, the manual analysis has shown that due to these linguistic challenges in tweets, the user of online mt tools may receive a fluent translation which deviates drastically from the sentiment of source in such a way that the reader would either understand the opposite sentiment or lose the sentiment all together. the error analysis presented in this paper, therefore, points to essential ethical issues that should be taken into consideration when adopting a fully automated translation technology to transfer users' stance on online platforms. we also touched upon the reliability of automatic quality measures for evaluating mt systems performance in transferring emotion. we have shown that the standard evaluation measures were not able to give a penalty proportional to the incorrect translation of emotion in a sample dataset of mistranslated tweets. this points to the fact that critical mistranslation of emotions by online mt systems may go undetected if the performance is gauged by conventional metrics such as bleu and meteor. we believe that evaluating the performance of mt systems in translating sentiment-oriented text is an under-recognised problem in mt research. future work should address the possibility of introducing a sentiment measure to reflect how far the mt system transfers the correct affective message in the source text as well as detect critical distortions of the source sentiment.