Sentiment-Aware Measure ( SAM ) for Evaluating Sentiment Transfer by Machine Translation Systems (2021)

Abstract: In translating text where sentiment is the main message, human translators give particular attention to sentiment-carrying words. The reason is that an incorrect translation of such words would miss the fundamental aspect of the source text, i.e. the author’s sentiment. In the online world, MT systems are extensively used to translate User-Generated Content (UGC) such as reviews, tweets, and social media posts, where the main message is often the author’s positive or negative attitude towards the topic of the text. It is important in such scenarios to accurately measure how far an MT system can be a reliable real-life utility in transferring the correct affect message. This paper tackles an under-recognized problem in the field of machine translation evaluation which is judging to what extent automatic metrics concur with the gold standard of human evaluation for a correct translation of sentiment. We evaluate the efficacy of conventional quality metrics in spotting a mistranslation of sentiment, especially when it is the sole error in the MT output. We propose a numerical “sentiment-closeness” measure appropriate for assessing the accuracy of a translated affect message in UGC text by an MT system. We will show that incorporating this sentiment-aware measure can significantly enhance the correlation of some available quality metrics with the human judgement of an accurate translation of sentiment.

## introduction standard quality measures for assessing the performance of machine translation systems, such as bleu @xcite , are domain agnos-tic; they evaluate the translation accuracy regardless of the semantic domain or linguistic peculiarities of the source text. consequently, they give equal penalty weight to inaccurate translation of n-grams, which may lead to performance overestimation (or underestimation). for example, the arabic high-rated goodreads book review showing the reviewers overall satisfaction of a novel: ' ('the novel is great, its only flaw is the last part') is mistranslated by the available online translation tool which wrongly outputs a negative sentiment ('the novel is terrible, its only flaw is the last part'). despite the distortion of the affect message, this translation receives an equally high score as a correct translation (the story is great, its only flaw is the last part), but which uses story instead of novel. this is because bleu mildly penalises the wrong translation as it swaps only one uni-gram ('great') with its opposite ('terrible'). yet, the mistranslation of this particular uni-gram is a critical error as it is most pivotal in transferring the sentiment, and hence the mt performance is over-estimated. there have been numerous efforts to address the common pitfalls of n-gram-based evaluation metrics by incorporating semantic and contextual features. however, despite research evidence of its analytical limitations, bleu is still the de facto standard for mt performance evaluation @xcite . moreover, although the introduction of more semantically-oriented metrics showed a better correlation with human judgement, still the estimation of sentiment preservation in ugc has not yet been investigated. this is par-tially due to the fact that the domain and linguistic style of the wmts datasets typically used for metric evaluation (e.g. newstest2020, canadian parliament, wikipedia, un corpus) is quite different than the non-standard noisy ugc where sentiment is the main content of a telegraphic message @xcite @xcite . assessed over these wmt datasets, some metrics manifest an almost perfect correlation with human evaluation on the segment-level (e.g. wmt20 participating metrics record results of up to 0.97 pearson correlation on the newstest2020) @xcite . however, research has also shown that metrics usually report weaker correlation with low human assessment score ranges @xcite . in this paper, we argue that the high correlation of some metrics may not be replicated with a different domain such as sentiment-oriented ugc, specifically when there is a mistranslation of sentiment-critical word(s). for this reason, we propose a 'sentiment-closeness' measure that can accommodate for a better evaluation of the mt system's ability to capture the correct sentiment of the source text. to present our sentiment quality measure, we first briefly explain in section 2 why metrics commonly used for mt quality estimation may not always be efficient in assessing sentiment-critical translation errors. in section 3, we present an experiment to quantitatively assess the divergence of the analysed metrics from a human judgement of a correct/incorrect translation of an affect message. section 4 presents our proposed solution for finetuning quality metrics for a better correlation with human judgement. section 5 presents the results of incorporating our sentiment-measure in different quality metrics. in section 6 we conduct an error analysis of our empirical approach and discuss its limitations. finally, section 7 presents a conclusion on our conducted experiments. ## available metrics automatic evaluation metrics usually take the output of an mt system (hypothesis) and compare it to one or several translations produced by human translator(s) (reference). based on their matching methods, the most commonly used automatic metrics can be broadly categorised into: surface ngram matching and embedding matching. surface n-gram methods work by calculating exact match-ing, heuristic matching or an edit distance between the aligned n-grams of the reference and hypothesis translation(s). the embedding methods, on the other hand, calculate a similarity score between learned token representations, such as contextual embedding vectors, with or without the aid of external linguistic tools. in the following sections, we briefly explain the methods behind three canonical metrics as representative of each category. we illustrate why the theoretical foundation of each metric may not be optimum for evaluating the translation of sentiment-oriented ugc. ## evaluation of mistranslated sentiment we have shown in the previous section that three canonical mt evaluation metrics do not give a penalty proportional to sentiment-critical errors on segment-level by an mt online tool. in order to quantify how far the aforementioned quality metrics diverge or correlate with human judgement of sentiment transfer, we measure the performance of each metric on a dataset of tweets that had sentiment-critical translation errors. to compile this data, first we used the twitter built-in translation system (google api) to translate a dataset of tweets annotated for sentiment foot_1 . the source dataset amounted to ≈7,000 tweets in three languages: english, arabic and spanish. we translated the arabic and spanish into english and the english was translated into spanish and arabic. to extract instances where the mt system failed to translate the sentiment correctly, we built an english sentiment-detection classifier by fine-tuning a roberta xml model @xcite on an english dataset of 23,000 tweets annotated for sentiment 3 . the english classifier was used to predict the sentiment of the google api output for the translation of the arabic and spanish tweets into english and the english back-translation of the english tweets translated into spanish and arabic. the classifier's predicted sentiment was compared to the gold standard emotion of the source text, and instances of discrepancy were extracted as potential mistranslations of sentiment. finally, from the extracted instances, we manually built a translation quality evaluation dataset. the quality evaluation dataset, henceforth qe, consisted of target tweets where the error is exclusively a mistranslation of the sentiment-carrying lexicon. in these tweets, the mistranslations either completely flip the sentiment polarity of the source tweet, similar to examples 3 and 4 above, or transfer the same polarity but with a mitigated sentiment tone. the tweets with exclusive sentiment translation errors amounted to 300 tweets. we also added 100 tweet/translation pairs where the mt system transfers the correct sentiment. reference translations of the qe dataset were created by native speakers of the respective source languages. essentially, the reference translations aimed at correcting chunks that caused a distortion of the affect message and retained as many of the hypothesis n-grams as possible to detect how far each metric is sensitive to sentiment mismatching and not to the mismatch in other non-sentiment carrying words. the translators were also asked to assign a score to each pair of source-hypothesis tweet, where 1 is the poorest sentiment transfer and 10 is best sentiment transfer. the average scores of annotators were taken as the final human score 4 . to quantify the ability of the three metrics explained in section 2 to assess the transfer of sentiment in the qe dataset, we compared their scores of the translation hypotheses with the human judgement scores for sentiment transfer on the segmentlevel. we followed the wmt standard methods for evaluating quality metrics and used absolute pearson correlation coefficient r and the kendall correlation coefficient |τ | to evaluate each metric's performance against the human judgement. figures 2 and 3 show heatmaps visualising the pearson and kendall correlation coefficients for the studied metrics and the human scores, respectively. as seen from figures 2 and 3 , both bertscore and meteor achieve a better correlation with the human judgement than bleu 5 which achieves only 0.16 and 0.12 pearson and kendall correlations, respectively. however, the relatively overall low correlations (max @xmath0.39 and max |τ | = 0.27) raise important doubts as to the reliability of these 4 annotators were computational linguists working on mt research. 5 we use the sacrebleu implementation of the bleu score for all the experiments @xcite . accepted metrics for ranking mt systems which translate sentiments. bearing in mind that 75% of the segments in the qe dataset have critical translation errors that seriously distort the sentiment, the low correlation results highlight the need for a sentiment-targeted measure that can improve a metric's efficacy in capturing mistranslated sentiment by an mt system in real-life scenarios. ## sentiment-aware measure (sam) for machine translated ugc in this section, we propose a new measure for assessing mt performance that takes into account the sentiment similarity between the mt system translation and the reference. this sentiment measure should be used as a fine-tuning tool to adjust a quality score in cases where it is used to assess the translation quality of sentiment-oriented text. the sam score is calculated by using the sentiword dictionary of prior polarities @xcite . sentiword is a sentiment lexicon that combines the high precision of manual lexica and the high coverage of automatic ones (covering 155,000 words). it is based on assigning a 'prior polarity' score for each lemma-pos in both sentiwordnet and a number of human-annotated sentiment lexica @xcite . the prior polarity is the out-of-context positive or negative score which a lemma-pos evokes. it is reached via an ensemble learning framework that combines several formulae where each lemma-pos is given the score that receives the highest number of votes from the different formulae. sentiword prior polarity scores have been proven to achieve the highest correlation with human scores in sentiment analysis regression and classification semeval tasks @xcite . we assume that our sentiment adjustment factor, sam, is proportional to the distance between the sentiment scores of the unmatched words in the system translation and the reference, the higher the distance the greater the sam adjustment. to calculate the sam score, we designate the number of remaining mismatched words in the system translation and reference translation by m and n, respectively. we calculate the total sentiword sentiment score for the lemma-pos of the mismatched words in the translation and reference sentences using a weighted average of the sentiment score of each mismatched lemma-pos. the weight of a hypothesis mismatched word w h and a reference mismatched word w r is calculated based on the sentiment score of its lemma-pos, s, as follows: then the total sentiment score for hypothesis s h and reference s r is given by: the normalised sam adjustment is given by: to illustrate how sam score adjusts a metric score with respect to the transfer of sentiment, as can be seen from ## assessing sam performance to check how far the sam measure can improve the evaluation of sentiment, we assessed the performance of the three chosen metrics on the qe dataset utilised in the experiment in section 3 with the sam adjustment. figures 4 and 5 show heatmaps of the pearson and kendall correlations of the sam adjusted metrics with the human judgement in the qe dataset. compared to metric scores without the sam adjustment, figures 4 and 5 show that the combination of sam with the three metrics consistently leads to an improvement in each metric's correlation with the human judgement of a correct sentiment transfer in our dataset. overall, bertscore records the highest pearson correlation coefficients and kendall rank dependence (0.59 and 0.44, respectively). this means that it is better able to penalise critical translation errors in our qe dataset. moreover, compared to their scores without sam, ## error analysis and limitations human sentiment can be expressed in intricately subtle ways so that the mistranslation of the affect message is not necessarily reflected in divergence of polarity scores. we conducted an error analysis on instances where the sam adjustment scores were not able to capture the mt's failure to transfer the correct sentiment due to different linguistic phenomena. the first phenomenon we identified is related to structure shifting. for example, the sentiment distance between the source tweet 'i was saddened by him' and its mistranslation 'i made him sad' is very small despite the flipping of sentiment direction in the translation. the three metrics with and without the sam adjustment failed to penalise this type of distortion in the qe dataset. second, some words in the lexicon did not have a score representative of their sentiment weight. for example, most prepositions in the sentiword lex-icon are neutral, yet by checking the data, it was found out that a mistranslation of a preposition can distort the affect message. for example, the source tweet 'what is the benefit of me in this world' was mistranslated by the mt system as 'what is the benefit for me in this world'; the wrong preposition causes the translation to lose the sad tone in the source tweet. again, similar instances were not adequately measured with the three metrics with and without the sam adjustment. third, some nuanced sentiment-carrying words specific for the informal style of tweets caused a mistranslation of sentiment which was not captured by our approach. for instance, a one-word tweet referring to a political figure as 'prick' was mistranslated as 'sting'. the source is a slang word used to refer to a mean, contemptible man. the translation wrongly received a high score since both 'prick' and 'sting' had very similar negative sentiment weights. the translation, however, fails to reflect the aggressive sentiment in the source tweet. the bleu metric, on the other hand, succeeded in giving a penalty to similar short mistranslated tweets without the sam adjustment. one other limitation to the current approach for assessing the translation of sentiment is that it relies on an english sentiment lexicon. the applicability of this approach to other languages depends on the availability of a similar high-precision and highcoverage sentiment lexicon. we have overcome this problem by using the english backtranslations of the arabic and spanish tweets in the qe dataset. there are, however, multilingual translations of english sentiment lexica that are commonly used in the sentiment-analysis of non-english text. it remains to be tested how far a translated sentiment lexicon is capable of measuring sentiment transfer among different language pairs by the sam scoring approach. ## conclusion the most frequent scenario in which an mt system is used to translate sentiment-oriented text is for the translation of online ugc such as tweets, reviews or social media posts. the users of the mt online tool take these translations at face-value as there is no human intervention for accuracy checking. it is important, therefore, to ensure the reliability of the mt system to accurately transfer the author's affect message before it is used as an online tool. the reliability of an mt system with such big data is commonly measured by automatic metrics. we have shown that conventionally accepted metrics may not always be an optimum solution for assessing the translation of sentiment-oriented ugc. we presented an empirical approach to quantify the notion of sentiment transfer by an mt system and, more concretely, to modify automatic metrics such that its mt ranking comes closer to a human judgement of a poor or good translation of sentiment. despite limitations to our approach, the sam adjustment serves as a proxy to the complicated task of manually evaluating the translation of sentiment across different languages.