BLEU , METEOR , BERTS core: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-Oriented Text (2021)

Abstract: Social media companies as well as censorship authorities make extensive use of artificial intelligence (AI) tools to monitor postings of hate speech, celebrations of violence or profanity. Since AI software requires massive volumes of data to train computers, automatic-translation of the online content is usually implemented to compensate for the scarcity of text in some languages. However, machine translation (MT) mistakes are a regular occurrence when translating sentiment-oriented user-generated content (UGC), especially when a low-resource language is involved. In such scenarios, the adequacy of the whole process relies on the assumption that the translation can be evaluated correctly. In this paper, we assess the ability of automatic quality metrics to detect critical machine translation errors which can cause serious misunderstanding of the affect message. We compare the performance of three canonical metrics on meaningless translations as compared to meaningful translations with a critical error that distorts the overall sentiment of the source text. We demonstrate the need for the fine-tuning of automatic metrics to make them more robust in detecting sentiment critical errors.

## introduction facebook has once apologised after its machine-translation service lead to an arrest of a man from the west bank whose profile posting in his native dialect that read "good morning" was mistranslated as "attack them", and later automatically detected by authorities as an incitement to violence 3 . the main danger in this type of mt error is that it changes the author's sentiment, here from positive to a negative or rather aggressive emotion. research on translation of sentiment by mt systems has shown that users encounter similar mistakes where the sentiment polarity of the source is flipped to its exact opposite due to a mistranslation of a contronym, a dialectical expression, or a missed negation marker, especially in translation of online content of low-resource languages @xcite . in machine translation research, the reliability of mt systems is conventionally measured by automatic quality metrics such as bleu @xcite and meteor @xcite . the aim of these automatic quality metrics is to evaluate a translation hypothesis (i.e. the automatic translation) against a reference translation, which is normally produced by a human translator. good evaluation metrics should have a high correlation with human judgement on the quality of translation. recently some automatic metrics have achieved a significant correlation with human judgement on the wmt metrics task datasets (see @xcite @xcite ). however, research has reported weaker correlation with low human assessment score ranges for segment-level evaluation @xcite . these findings point to the challenges involved in detecting low-quality translations by automatic metrics. in this work, we focus on the problem of evaluating critical translation errors that can cause serious misunderstanding of the sentiment conveyed in the source text. to illustrate this point, suppose we are evaluating the mt output "people are dead, starving in your presence, may god forgive you" with its reference "people are dead, starving in your presence, may god not forgive you" 4 . the error in the mt output is only the missing of the word not, however, this omission causes the translation to convey the exact opposite sentiment of the source. we argue that such translation errors should be considered more critical than those which produce ungrammatical or low-quality translations, but do not significantly distort the message of the source. however, as we show in this paper, automatic quality metrics fail to give a penalty to this type of critical error proportional to its gravity and may equate this hypothesis with another that also has a uni-gram mistake, but transfers the affect message (e.g people are dead, hungry in your presence, may god not forgive you). in this research we conduct an experiment with three canonical automatic quality metrics, bleu, meteor and bertscore @xcite . we measure the ability of each metric to penalise sentiment critical errors that severely distort the affect message as compared to translations which correctly transfer the correct sentiment as well as mistranslations that produce incomprehensible content in the target language. we first briefly present the three metrics in section 2. then, in section 3, we explain our experiment and summarise the results. in section 4, we give our concluding remarks. ## related work the standard metric for assessing empirical improvement of mt systems is bleu. simply stated, the objective of bleu is to compare n-grams of the candidate translation with n-grams of the reference translation and count the number of matches; the more the matches, the better the candidate translation. the final score is a modified n-gram precision multiplied by a brevity penalty to account for both frequency and adequacy. due to its restrictive exact matching to the reference, bleu does not accommodate for importance n-gram weighting which may be essential in assessing a sentiment-critical error. however, despite research evidence of its analytical limitations @xcite , bleu, is still the de facto standard for mt performance evaluation because it is easy to calculate regardless of the languages involved. meteor, on the other hand, incorporates semantic information as it evaluates translation by calculating either exact match, stem match, or synonymy match. for synonym matching, it utilises wordnet synsets @xcite . more recent versions (meteor 1.5 and meteor++2.0) apply importance weighting by giving smaller weight to function words @xcite . however, the meteor weighting scheme would not allow for a great penalty of the missing negation marker in the hypothesis of our example above. in fact, the meteor score for twitter's mt wrong translation is 0.91, whereas the score for the correct translation (people are dead, starving in your presence, may god not forgive you) is 0.99. the main culprit for this proportionally inaccurate scoring is the function word weighting which causes the metric to be over permissive despite the mt engine missing of a negation marker crucial to the sentiment of the source tweet. both meteor and bleu assess the quality of translation in terms of surface n-gram matching between the mt output and a human reference(s). after the introduction of pretrained contextual word models, there has been a recent trend to use large-scale models like bert @xcite for mt evaluation to incorporate semantic contextual information of tokens in comparing translation and reference segments. a number of embedding-based metrics has proven to achieve the highest performance in recent wmt shared tasks for quality metrics (e.g. @xcite @xcite ). we take bertscore as representative of this category. bertscore computes a score based on a pair wise cosine similarity between the bert contextual embeddings of the individual tokens for the hypothesis and the reference. accordingly, a bertscore close to 1 indicates proximity in vector space and hence a good translation. in the following section, we explain our experiment for assessing the performance of these three metrics with respect to critical translation errors that seriously distort the affect message of the source. ## conclusion in this research, we conducted an experiment with three canonical automatic quality metrics to evaluate their ability to penalise a critical translation error that seriously distorts the affect message of the source text. the average segmentlevel scores for the three metrics showed that sentiment-critical and non-critical errors are not appropriately distinguishable especially in our synthetic dataset. this shows that in scenarios where the mt output is an exact match to the reference except for one sentiment-pivotal word, the automatic quality metric becomes less sensitive to the mistranslation error. similarly, with the authentic datasets, the average scores for meteor showed that mistranslations with one or two critical errors are not appropriately penalised. moreover, with both the authentic and synthetic data, the relatively high inconsistency of segment-level scores for hypotheses with one or two sentiment-critical errors suggests that a distortion of the sentiment content may misleadingly receive high scores by any of the three metrics. the results of the experiment call attention to the need for a sentiment-targeted evaluation measure that can adequately assess this type of critical translation errors that have can serious consequences in determining the sentiment stance of the author. our future work will focus on fine-tuning the quality metrics to capture sentiment-critical lexicon to improve its performance with sentiment-oriented text.