Is it Great or Terrible? Preserving Sentiment in Neural Machine Translation of A rabic Reviews (2020)

Abstract: Since the advent of Neural Machine Translation (NMT) approaches there has been a tremendous improvement in the quality of automatic translation. However, NMT output still lacks accuracy in some low-resource languages and sometimes makes major errors that need extensive postediting. This is particularly noticeable with texts that do not follow common lexico-grammatical standards, such as user generated content (UGC). In this paper we investigate the challenges involved in translating book reviews from Arabic into English, with particular focus on the errors that lead to incorrect translation of sentiment polarity. Our study points to the special characteristics of Arabic UGC, examines the sentiment transfer errors made by Google Translate of Arabic UGC to English, analyzes why the problem occurs, and proposes an error typology specific of the translation of Arabic UGC. Our analysis shows that the output of online translation tools of Arabic UGC can either fail to transfer the sentiment at all by producing a neutral target text, or completely flips the sentiment polarity of the target word or phrase and hence delivers a wrong affect message. We address this problem by fine-tuning an NMT model with respect to sentiment polarity showing that this approach can significantly help with correcting sentiment errors detected in the online translation of Arabic UGC.

## introduction translation of user generated content (ugc) such as user reviews is becoming common on multilingual websites which sell products and services such as amazon.com or booking.com. in this context, sentiment preservation in automatic machine translation (these days usually neural machine translation (nmt) output) is of great importance because many decisions about purchasing a product or service are based on the comments made by others. there have been different studies which explored the transfer of sentiment in mt, but most of these studies assess how far automatic sentiment classification systems can capture sentiment information from the translations @xcite @xcite . the objective of most research in this area is from a sentiment classification perspective rather than a translation accuracy perspective. hence, it measures how far automatic translation of a language into english can help with the sentiment classification of that language by applying the available english sentiment resources on the target text @xcite @xcite . this study is concerned with nmt accuracy of sentiment transfer at the word/phrase level and shows that inaccurate translation can transfer a completely opposite affect message. moreover, the translation of ugc such as product reviews constitutes a significant challenge for nmt online tools in general and for arabic ugc in particular. the reason is that arabic ugc is usually a mix of dialectical arabic (da) and modern standard arabic (msa) which differ significantly on the lexico-grammatical level. the same word or phrase can have opposite sentiment polarities in the two versions of the arabic language, which often leads to a mistranslation of the sentiment message. if the nmt engine is robust enough to handle this type of code-switching, it can become more reliable not only in downstream nlp tasks such as cross-lingual information retrieval, but also in real-life scenarios when internet users resort to online translation tools to check the reviews of a particular product of interest. in this study, we assess the degree to which the nmt online tools transfer sentiment accurately at the word/phrase level and suggest methods for improving the accuracy of the translation of sentiment in arabic ugc. we aim to answer the following questions: to answer the above research questions, this paper is divided as follows: section 2 presents related work on sentiment transfer in mt. section 3 analyzes sentiment translation errors of nmt online tools of arabic reviews and provides a qualitative typology of most frequent error types. in section 4, we present different methodological approaches for correcting the nmt online sentiment transfer errors. section 5 provides task-specific evaluation metrics for assessing the sentiment accuracy improvement by the proposed methods. section 6 presents a conclusion on the different experiments as well as limitations of the present study. ## related work research on the translation of sentiment in mt has focused on the idea that despite significant errors in sentiment transfer, automatic sentiment classification systems are still able to capture sentiment information from the translations @xcite @xcite . @xcite showed that although certain attributes of automatically translated text 'may mislead humans' with regards to the true sentiment of the source text, they do not seem to affect the automatic sentiment analysis systems @xcite . the rationale behind these studies is that if we have a good machine translation model, it will eliminate the necessity to develop sentiment analysis resources specific of the source language @xcite . given the proliferation of english sentiment analysis tools, we can always make use of them by conducting sentiment analysis on the english translation of the source text, even if the translation is not of high quality @xcite . studies also show that developed mt models, as well as online translation tools such as google translate and microsoft translate, can be relied upon to perform sentiment classification of the target text despite any accuracy errors @xcite . this is because sentiment classification systems can learn an appropriate model even from mistranslated text -especially when automatic translation makes consistent errors @xcite . moreover, statistically, studies of sentiment translation have shown that automatic translation leads to only about 60% match with manually annotated sentiment labels. yet, automatic sentiment classifiers can still perform well despite these errors which can markedly impact human perception of sentiment in the source tweet/review @xcite . recently, mt studies started to tackle how sentiment can be preserved in the translation of ugc from a translation accuracy perspective. @xcite show that back translation of restaurant reviews can provide significant improvement over existing online systems particularly in preserving sentiment of translated ugc. they translate a large corpus of reviews from the target language into english and then use it in model training. they use domain tags at the training stage to distinguish user-generated source text. their results prove that both synthetic data and domain tags can achieve good results in preserving the affect polarity on the sentence level. while their model is promising, they still point to serious errors in the translation of ugc such as missing negations, hallucinations, unrecognized named entities and insensitivity to context. they suggest that this task is far from solved @xcite . @xcite makes an attempt to improve the sentiment transfer of translated tweets. they show that freely available translation tools often cause the sentiment encoded in the original tweet to be altered. as a consequence, they build separate negative, neutral and positive sentiment smt models to improve sentiment preservation in the target language. they show that a translation model specific of each sentiment pole provides much better results over a single baseline model trained on the whole twitter data, regardless of the sentiment class. they attempt to strike a balance between improving sentiment transfer and preserving translation accuracy as measured by evaluative metrics such as bleu and meteor @xcite . a similar technique is used by @xcite as they build a valence sensitive nmt model for the translation of ambiguous words that can have different polarities in different contexts. each input sentence is annotated with a positive or negative label to indicate its polarity. they show that adding this tag to the source sentence at the training time and creating dual polarity embedding vectors for ambiguous words can improve sentiment transfer at the word level @xcite . there has also been some research on finding alternative means for assessing the transfer of sentiment in mt other than the typical accuracy metrics. @xcite show that automatic evaluation metrics such as bleu and meteor tend to neglect sentiment discrepancies between source and target output. they suggest assessing the accuracy of sentiment preservation by targeted metrics that measure how well polysemous words are translated, or how well sentiments expressed in the original text can be recovered from its translation @xcite . to assess sentiment preservation in mt, @xcite use a sentiment lexicon-based measure in combination with regular evaluation metrics such as the bleu score. several studies also resort to human evaluation to measure how far a model improves sentiment transfer at the word/phrase level @xcite . in this study, we evaluate the preservation of sentiment in translation not as a sentiment classification task, but from a translation accuracy perspective. we show that translation inaccuracies at the word/phrase level can seriously impact the transfer of sentiment in arabic ugc, which can lead to problems for users of the mt tools in real-life situations. several commercial global platforms rely on publicly available mt engines to translate product reviews into the customers own language to facilitate communication between partners and customers foot_0 . inaccurate translation of reviewers' sentiment would defeat the purpose of using such tools. moreover, in commercial situations, companies may want to find out what their users think of particular products so the accuracy of each translation review counts. broadly speaking, online tools such as google translate, are commonly utilized as an off-the-shelf solution for the translation of ugc in arabic as well as in other languages. error-analysis of sentiment translation by online tools, however, has proved that the true sentiment of arabic reviews can be either missed or flipped to its exact opposite pole. ## error analysis in order to measure how accurately nmt online tools transfer sentiment of arabic ugc, we chose a dataset of book reviews scraped from goodreads foot_1 @xcite . each review has a rating between 1-5 assigned by its author. the language of the reviews is a mix of msa and da, with the largest majority of da reviews in the egyptian dialect. reviews in the dataset are of varying lengths, but a large number of them have over than 100 tokens. long reviews were split to a maximum of 20 tokens per review. after splitting, the data amounted to about 230,000 sentences. this dataset was translated into english using the google translate api and was analysed using both manual and automatic error analysis, focusing on mistranslation of sentiment. automatic sentiment analysis tools were utilized to detect sentiment errors in the dataset and subsequently select a sample for manual error analysis. since the main objective is to assess the accuracy of sentiment translation at the word level, we used an automated lexicon-based sentiment measure on the google translate output. we applied the cloudbased microsoft azure text analytics tools for sentiment analysis 3 on around 13,000 target sentences. the azure's sentiment analysis api generates sentiment scores using classification features such as n-gram sentiment scores, part-of-speech tags and word embeddings. it evaluates text and returns a label (positive, neutral, negative) for each sentence as well as numeric confidence scores that range from 0 to 1 for each sentiment category. scores closer to 1 indicate a higher confidence in the label's classification, while lower scores indicate lower confidence. for each sentence, the predicted scores associated with the labels (positive, negative and neutral) add up to 1. following traditional methods in sentiment classification @xcite , we used the rating of the book review as indicative of its sentiment polarity and compared it to the confidence scores generated by the azure sentiment analysis api. accordingly, reviews were categorized based on discrepancies between the ratings and the confidence scores. a positive review that had a rating of 4 or above and an english negative sentiment score of 0.5 and above was extracted as an example of potential wrong negative polarity in the target text. similarly, reviews with negative ratings of 2 and below and a positive english sentiment score of 0.5 and above were extracted as instances of potential wrong positive polarity in the target text. this amounted to a total of around 4,000 potentially negative sentiment errors and around 2,000 of potentially positive sentiment errors. a sample of reviews of 1000 parallel sentences from the dataset that had discrepancies between the automatic sentiment score and the review rating were manually analyzed to detect reasons for these discrepancies. by analyzing the causes of mistranslation of sentiment in this sample, the mistakes were categorized into a five group typology. the typology of sentiment translation errors are summarized in the following sections. one or two examples for each type of errors will be mentioned in the following sections. the table in appendix a gives more examples of each type. ## notes on error typology in order to get an indication of the frequency of each type of errors in the whole dataset, words/phrases belonging to each group were extracted based on their frequency in the corpus then the frequency of their mistranslation in the dataset was manually calculated. figure 1 shows the frequency of the mistranslation instances in the total reviews dataset of the extracted words as representative of each type of errors. as can be seen almost all the instances of the most frequent idiomatic and dialectical expressions are mistranslated. moreover, around 65% of the time the positive meaning of contronyms was flipped to negative. consequently, there were sentimentally incongruous terms where a positive noun was described with a highly improbable negative adjective (e.g. horrible achievement, terrible masterpiece, terrible happiness, and so on). our error typology showed that more than one type of errors is due to code-switching between da and msa in arabic ugc. there have been several approaches to tackle the challenges of translating arabic da such as paraphrasing source text in msa before translation @xcite . other studies have also proved that concatenating small amount of arabic dialectical data can significantly improve the translation quality @xcite . however, our error analysis has shown that even in msa context the sentiment of words/phrases can be mistranslated. pivoting on msa can solve straightforward problems such as dialectical phrases and dialectical idioms. however, preserving the sentiment of the source text would require addressing the polarity of words with opposite meanings such as contronyms. since it is beyond the scope of this paper to tackle all the error types, an attempt was made to address the problem of contronyms in arabic ugc. in this paper we propose a sentimentsensitive nmt model that is robust to the opposite sentiment polarities of arabic contronyms either due to code-switching between da and msa or to contextual variations in arabic ugc. details of the experiment are explained in the following sections. ## sentiment oriented nmt system in order to improve the translation of contronyms in arabic ugc, we propose two transformer (nmt) models infused with sentiment information at the encoding stage. we show that training on a sentiment oriented small-sized data can provide high performance results in preserving the sentiment of challenging contronyms in arabic ugc. details of data preprocessing and model architectures are explained in the following sections. ## conclusion this study has shown that arabic ugc has peculiar qualities which constitute a challenge to automatic translation tools especially in its ability to preserve the sentiment message. an error typology was derived after analysing the data. this typology has highlighted how sentiment errors can impair the translation of sentiment-oriented arabic ugc such as product reviews. since automatic online translation tools are heavily relied upon by users and commercial platforms to translate reviews, it is of essential importance to fine-tune nmt models to the correct sentiment message in the source text. moreover, it has been common practice for nmt training to use big parallel data which involves very high computational power and requires availability of large authentic data. the proposed nmt models in this study, however, showed that infusing contextual cues at the training stage of a relatively small data can improve the translation of sentiment in arabic ugc both on the word and sentence level. this approach can help in providing greener training and make it feasible to construct competitive nmt tools for low-resource domains such as arabic ugc. moreover, we showed that translation quality metrics of sentimentoriented arabic ugc needs to be supplemented with other metrics that assess the preservation of sentiment. we proposed lexicon-based metrics that take into account the sentiment score of single words as well as their context. finally, this study has tackled one of several challenges in the translation of sentiment in arabic ugc. future research will address the whole spectrum of challenges to improve the accuracy of sentiment preservation which is of vital importance in the translation of arabic ugc.